Cleanup Features - Updating convert command - Fix Download manager

For conflicts, I think the mention hint "This should be modified because it mentions ..." is missing.

This PR makes a number of changes:

# Updating `Features`

Features are a complex mechanism provided in `tfds` to be able to modify a dataset on-the-fly when serializing to disk and when loading from disk.

We don't really need this because (1) it hides too much from the user and (2) our datatype can be directly mapped to Arrow tables on drive so we usually don't need to change the format before/after serialization.

This PR extracts and refactors these features in a single `features.py` files. It still keep a number of features classes for easy compatibility with tfds, namely the `Sequence`, `Tensor`, `ClassLabel` and `Translation` features.

Some more complex features involving a pre-processing on-the-fly during serialization are kept:
- `ClassLabel` which are able to convert from label strings to integers,
- `Translation`which does some check on the languages.

# Updating the `convert` command

We do a few updates here
- following the simplification of the `features` (cf above), conversion are updated
- we also makes it simpler to convert a single file
- some code need to be fixed manually after conversion (e.g. to remove some encoding processing in former tfds `Text` features. We highlight this code with a "git merge conflict" style syntax for easy manual fixing.

# Fix download manager iterator

You kept me up quite late on Tuesday night with this `os.scandir` change @lhoestq ;-)


remove boto3 and promise dependencies

With the new download manager, we don't need `promise` anymore.
I also removed `boto3` as in [this pr](https://github.com/huggingface/transformers/pull/3968)


Replace tf.constant for TF

Replace simple tf.constant type of Tensor to tf.ragged.constant which allows to have examples of different size in a tf.data.Dataset.

Now the training works with TF. Here the same example than for the PT in collab:

```python
import tensorflow as tf
import nlp
from transformers import BertTokenizerFast, TFBertForQuestionAnswering

# Load our training dataset and tokenizer
train_dataset = nlp.load('squad', split="train[:1%]")
tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')

def get_correct_alignement(context, answer):
    start_idx = answer['answer_start'][0]
    text = answer['text'][0]
    end_idx = start_idx + len(text)
    if context[start_idx:end_idx] == text:
        return start_idx, end_idx       # When the gold label position is good
    elif context[start_idx-1:end_idx-1] == text:
        return start_idx-1, end_idx-1   # When the gold label is off by one character
    elif context[start_idx-2:end_idx-2] == text:
        return start_idx-2, end_idx-2   # When the gold label is off by two character
    else:
        raise ValueError()

# Tokenize our training dataset
def convert_to_features(example_batch):
    # Tokenize contexts and questions (as pairs of inputs)
    input_pairs = list(zip(example_batch['context'], example_batch['question']))
    encodings = tokenizer.batch_encode_plus(input_pairs, pad_to_max_length=True)

    # Compute start and end tokens for labels using Transformers's fast tokenizers alignement methods.
    start_positions, end_positions = [], []
    for i, (context, answer) in enumerate(zip(example_batch['context'], example_batch['answers'])):
        start_idx, end_idx = get_correct_alignement(context, answer)
        start_positions.append([encodings.char_to_token(i, start_idx)])
        end_positions.append([encodings.char_to_token(i, end_idx-1)])

    if start_positions and end_positions:
      encodings.update({'start_positions': start_positions,
                        'end_positions': end_positions})
    return encodings

train_dataset = train_dataset.map(convert_to_features, batched=True)

columns = ['input_ids', 'token_type_ids', 'attention_mask', 'start_positions', 'end_positions']
train_dataset.set_format(type='tensorflow', columns=columns)
features = {x: train_dataset[x] for x in columns[:3]}
labels = {"output_1": train_dataset["start_positions"]}
labels["output_2"] = train_dataset["end_positions"]
tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)
model = TFBertForQuestionAnswering.from_pretrained("bert-base-cased")
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.NONE, from_logits=True)
opt = tf.keras.optimizers.Adam(learning_rate=3e-5)
model.compile(optimizer=opt,
              loss={'output_1': loss_fn, 'output_2': loss_fn},
              loss_weights={'output_1': 1., 'output_2': 1.},
              metrics=['accuracy'])
model.fit(tfdataset, epochs=1, steps_per_epoch=3)
```


Updating caching mechanism - Allow dependency in dataset processing scripts - Fix style and quality in the repo

This PR has a lot of content (might be hard to review, sorry, in particular because I fixed the style in the repo at the same time).

# Style & quality:
You can now install the style and quality tools with `pip install -e .[quality]`. This will install black, the compatible version of sort and flake8.
You can then clean the style and check the quality before merging your PR with:
```bash
make style
make quality
```

# Allow dependencies in dataset processing scripts
We can now allow (some level) of imports in dataset processing scripts (in addition to PyPi imports).
Namely, you can do the two following things:

Import from a relative path to a file in the same folder as the dataset processing script:
```python
import .c4_utils
```

Or import from a relative path to a file in a folder/archive/github repo to which you provide an URL after the import state with `# From: [URL]`:
```python
import .clicr.dataset_code.build_json_dataset  # From: https://github.com/clips/clicr
```

In both these cases, after downloading the main dataset processing script, we will identify the location of these dependencies, download them and copy them in the dataset processing script folder.

Note that only direct import in the dataset processing script will be handled.
We don't recursively explore the additional import to download further files.
Also, when we download from an additional directory (in the second case above), we recursively add `__init__.py` to all the sub-folder so you can import from them.

This part is still tested for now. If you've seen datasets which required external utilities, tell me and I can test it.

# Update the cache to have a better local structure

The local structure in the `src/datasets` folder is now: `src/datasets/DATASET_NAME/DATASET_HASH/*`

The hash is computed from the full code of the dataset processing script as well as all the local and downloaded dependencies as mentioned above. This way if you change some code in a utility related to your dataset, a new hash should be computed.


Add Pandas as format type

As detailed in the title ^^


create our own DownloadManager

The new download manager is ready. I removed the old folder and I fixed a few remaining dependencies.
I tested it on squad and a few others from the dataset folder and it works fine.

The only impact of these changes is that it breaks the `download_and_prepare` script that was used to register the checksums when we create a dataset, as the checksum logic is not implemented.

Let me know if you have remarks

Ok merged it (a bit fast for you to update the copyright, now I see that. but it's ok, we'll do a pass on these doc/copyright before releasing anyway)

Actually two additional things here @lhoestq (I merged too fast sorry, let's make a new PR for additional developments):
- I think we can remove some dependencies now (e.g. `promises`) in setup.py, can you have a look?
- also, I think we can remove the boto3 dependency like here: https://github.com/huggingface/transformers/pull/3968

I tried to create our own - and way simpler - download manager, by replacing all the complicated stuff with our own `cached_path` solution.
With this implementation, I tried `dataset = nlp.load('squad')` and it seems to work fine.

For the implementation, what I did exactly:
- I copied the old download manager
- I removed all the dependences to the old `download` files
- I replaced all the download + extract calls by calls to `cached_path`
- I removed unused parameters (extract_dir, compute_stats) (maybe compute_stats could be re-added later if we want to compute stats...)
- I left some functions unimplemented for now. We will probably have to implement them because they are used by some datasets scripts (download_kaggle_data, iter_archive) or because we may need them at some point (download_checksums, _record_sizes_checksums)

Let me know if you think that this is going the right direction or if you have remarks.
Note: I didn't write any test yet as I wanted to read your remarks first


[Tests] General Test Design for all dataset scripts

> I think I'm fine with this.
>
> The alternative would be to host a small subset of the dataset on the S3 together with the testing script. But I think having all (test file creation + actual tests) in one file is actually quite convenient.
>
> Good for me!
>
> One question though, will we have to create one test file for each of the 100+ datasets or could we make some automatic conversion from tfds dataset test files?

I think if we go the way shown in the PR we would have to create one test file for each of the 100+ datasets.

As far as I know the tfds test files all rely on the user having created a special download folder structure in `tensorflow-datasets/tensorflow_datasets/testing/test_data/fake_examples`.

My hypothesis was:
Becasue, we don't want to work with PRs, no `dataset_script` is going to be in the official repo, so no `dataset_script_test` can be in the repo either. Therefore we can also not have any "fake" test folder structure in the repo.

**BUT:** As you mentioned @thom, we could have a fake data structure on AWS. To add a test the user has to upload multiple small test files when uploading his data set script.

So for a cli this could look like:
`python nlp-cli upload <data_set_script> --testfiles <relative path to test file 1> <relative path to test file 2> ...`

or even easier if the user just creates the dataset folder with the script inside and the testing folder structure, then the API could look like:

`python nlp-cli upload <path/to/dataset/folder>`

and the dataset folder would look like
```
squad
- squad.py
- fake_data  # this dir would have to have the exact same structure we get when downloading from the official squad data url
```

This way I think we wouldn't even need any test files at all for each dataset script. For special datasets like `c4` or `wikipedia` we could then allow to optionally upload another test script.
We just assume that this is our downloaded `url` and check all functionality from there.

Thinking a bit more about this solution sounds a) much less work and b) even easier for the user.

A small problem I see here though:
1) What do we do when the depending on the config name the downloaded folder structure is very different? I think for each dataset config name we should have one test, which could correspond to one "fake" folder structure on AWS

@thomwolf What do you think? I would actually go for this solution instead now.
@mariamabarham You have written many more tfds dataset scripts and tests than I have - what do you think?

Regarding the tfds tests, I don't really see a point in keeping them because:

1) If you provide a fake data structure, IMO there is no need for each dataset to have an individual test file because (I think) most datasets have the same functions `_split_generators` and `_generate_examples` for which you can just test the functionality in a common test file. For special functions like these beam / pipeline functionality you probably need an extra test file. But @mariamabarham I think you have seen more than I have here as well

2) The dataset test design is very much intertwined with the download manager design and contains a lot of code. I would like to seperate the tests into a) tests for downloading in general b) tests for post download data set pre-processing. Since we are going to change the download code anyways quite a lot, my plan was to focus on b) first.

I like the idea of having a fake data folder on S3. I have seen datasets with nested compressed files structures that would be tedious to generate with code. And for users it is probably easier to create a fake data folder by taking a subset of the actual data, and then upload it as you said.

> > I think I'm fine with this.
> > The alternative would be to host a small subset of the dataset on the S3 together with the testing script. But I think having all (test file creation + actual tests) in one file is actually quite convenient.
> > Good for me!
> > One question though, will we have to create one test file for each of the 100+ datasets or could we make some automatic conversion from tfds dataset test files?
>
> I think if we go the way shown in the PR we would have to create one test file for each of the 100+ datasets.
>
> As far as I know the tfds test files all rely on the user having created a special download folder structure in `tensorflow-datasets/tensorflow_datasets/testing/test_data/fake_examples`.
>
> My hypothesis was:
> Becasue, we don't want to work with PRs, no `dataset_script` is going to be in the official repo, so no `dataset_script_test` can be in the repo either. Therefore we can also not have any "fake" test folder structure in the repo.
>
> **BUT:** As you mentioned @thom, we could have a fake data structure on AWS. To add a test the user has to upload multiple small test files when uploading his data set script.
>
> So for a cli this could look like:
> `python nlp-cli upload <data_set_script> --testfiles <relative path to test file 1> <relative path to test file 2> ...`
>
> or even easier if the user just creates the dataset folder with the script inside and the testing folder structure, then the API could look like:
>
> `python nlp-cli upload <path/to/dataset/folder>`
>
> and the dataset folder would look like
>
> ```
> squad
> - squad.py
> - fake_data  # this dir would have to have the exact same structure we get when downloading from the official squad data url
> ```
>
> This way I think we wouldn't even need any test files at all for each dataset script. For special datasets like `c4` or `wikipedia` we could then allow to optionally upload another test script.
> We just assume that this is our downloaded `url` and check all functionality from there.
>
> Thinking a bit more about this solution sounds a) much less work and b) even easier for the user.
>
> A small problem I see here though:
>
> 1. What do we do when the depending on the config name the downloaded folder structure is very different? I think for each dataset config name we should have one test, which could correspond to one "fake" folder structure on AWS
>
> @thomwolf What do you think? I would actually go for this solution instead now.
> @mariamabarham You have written many more tfds dataset scripts and tests than I have - what do you think?

I'm agreed with you just one thing, for some dataset like glue or xtreme you may have multiple datasets in it. so  I think a good way is to have one main fake folder and a subdirectory for each dataset inside

> Regarding the tfds tests, I don't really see a point in keeping them because:
>
> 1. If you provide a fake data structure, IMO there is no need for each dataset to have an individual test file because (I think) most datasets have the same functions `_split_generators` and `_generate_examples` for which you can just test the functionality in a common test file. For special functions like these beam / pipeline functionality you probably need an extra test file. But @mariamabarham I think you have seen more than I have here as well
> 2. The dataset test design is very much intertwined with the download manager design and contains a lot of code. I would like to seperate the tests into a) tests for downloading in general b) tests for post download data set pre-processing. Since we are going to change the download code anyways quite a lot, my plan was to focus on b) first.

For _split_generator, yes. But I'm not sure for _generate_examples because there is lots of things that should be taken into account such as feature names and  types, data format (json, jsonl,  csv, tsv,..)

Sounds good to me!

When testing, we could thus just override the prefix in the URL inside the download manager to have them point to the test directory on our S3.

Cc @lhoestq

Ok, here is a second draft for the testing structure.

I think the big difficulty here is "How can you generate tests on the fly from a given dataset name, *e.g.* `squad`"?

So, this morning I did some research on "parameterized testing" and pure `unittest` or `pytest` didn't work very well.
I found the lib https://github.com/wolever/parameterized, which works very nicely for our use case I think.
@thomwolf - would it be ok to have a dependence on this lib for `nlp`? It seems like a light-weight lib to me.

This lib allows to add a `parameterization` decorator to a `unittest.TestCase` class so that the class can be instantiated for multiple different arguments (which are the dataset names `squad` etc. in our case).

What I like about this lib is that one only has to add the decorator and the each of the parameterized tests are shown, like this:

![Screenshot from 2020-04-24 15-13-14](https://user-images.githubusercontent.com/23423619/80216326-2bd9a680-863e-11ea-8a0f-460976f5309c.png)

With this structure we would only have to upload the dummy data for each dataset and would not require a specific testing file.

What do you think @thomwolf @mariamabarham @lhoestq ?

I think this is a nice solution.

Do you think we could have the `parametrized` dependency in a `[test]` optional installation of `setup.py`? I would really like to keep the dependencies of the standard installation as small as possible.

> I think this is a nice solution.
>
> Do you think we could have the `parametrized` dependency in a `[test]` optional installation of `setup.py`? I would really like to keep the dependencies of the standard installation as small as possible.

Yes definitely!

UPDATE:

This test design is ready now. I added dummy data to S3 for the dataests: `squad, crime_and_punish, sentiment140` . The structure can be seen on `https://s3.console.aws.amazon.com/s3/buckets/datasets.huggingface.co/nlp/squad/dummy/?region=us-east-1&tab=overview` for `squad`.

All dummy data files have to be in .zip format and called `dummy_data.zip`. The zip file should thereby have the exact same folder structure one gets from downloading the "real" data url(s).

To show how the .zip file looks like for the added datasets, I added the folder `nlp/datasets/dummy_data` in this PR. I think we can leave for the moment so that people can see better how to add dummy data tests and later delete it like `nlp/datasets/nlp`.

The general idea is similar to how testing is done in `transformers`. There is one general `test_dataset_common.py` file which has a `DatasetTesterMixin` class. This class implements all of the logic that can be used in a generic way for all dataset classes. The idea is to keep each individual dataset test file as minimal as possible.

In order to test whether the specific data set class can download the data and generate the examples **without** downloading the actual data all the time, a MockDataLoaderManager class is used which receives a `mock_folder_structure_fn` function from each individual dataset test file that create "fake" data and which returns the same folder structure that would have been created when using the real data downloader.


[Download] Only create dir if not already exist

This was quite annoying to find out :D.
Some datasets have save in the same directory. So we should only create a new directory if it doesn't already exist.


[Make style]

I think this can be quickly reproduced.
I use `black, version 19.10b0`.

When running:
`black nlp/src/arrow_reader.py`
it gives me:

```
error: cannot format /home/patrick/hugging_face/nlp/src/nlp/arrow_reader.py: cannot use --safe with this file; failed to parse source file.  AST error message: invalid syntax (<unknown>, line 78)
Oh no! 💥 💔 💥
1 file failed to reformat.
```

The line in question is:
https://github.com/huggingface/nlp/blob/6922a16705e61f9e31a365f2606090b84d49241f/src/nlp/arrow_reader.py#L78

What is weird is that the trainer file in `transformers` has more or less the same syntax and black does not fail there:
https://github.com/huggingface/transformers/blob/cb3c2212c79d7ff0a4a4e84c3db48371ecc1c15d/src/transformers/trainer.py#L95

I googled quite a bit about black & typing hints yesterday and didn't find anything useful.
Any ideas @thomwolf @julien-c @LysandreJik ?

> I think this can be quickly reproduced.
> I use `black, version 19.10b0`.
>
> When running:
> `black nlp/src/arrow_reader.py`
> it gives me:
>
> ```
> error: cannot format /home/patrick/hugging_face/nlp/src/nlp/arrow_reader.py: cannot use --safe with this file; failed to parse source file.  AST error message: invalid syntax (<unknown>, line 78)
> Oh no! 💥 💔 💥
> 1 file failed to reformat.
> ```
>
> The line in question is:
> https://github.com/huggingface/nlp/blob/6922a16705e61f9e31a365f2606090b84d49241f/src/nlp/arrow_reader.py#L78
>
> What is weird is that the trainer file in `transformers` has more or less the same syntax and black does not fail there:
> https://github.com/huggingface/transformers/blob/cb3c2212c79d7ff0a4a4e84c3db48371ecc1c15d/src/transformers/trainer.py#L95
>
> I googled quite a bit about black & typing hints yesterday and didn't find anything useful.
> Any ideas @thomwolf @julien-c @LysandreJik ?

Ok I found the problem. It was the one Julien mentioned and has nothing to do with this line. Black's error message is a bit misleading here, I guess

Ok, just had to remove the python 2 syntax comments `# type`.

Good to merge for me now @thomwolf

Added Makefile and applied make style to all.
make style runs the following code:

```
style:
          black --line-length 119 --target-version py35 src
          isort --recursive src
```

It's the same code that is run in `transformers`.


[Map Function] add assert statement if map function does not return dict or None

Also added to an assert statement that if a dict is returned by function, all values of `dicts` are `lists`

Wait to merge until `make style` is set in place.

Updated the assert statements. Played around with multiple cases and it should be good now IMO.

IMO, if a function is provided that is not a print statement (-> returns variable of type `None`) or a function that updates the datasets (-> returns variable of type `dict`), then a `TypeError` should be raised.

Not sure whether you had cases in mind where the user should do something else @thomwolf , but I think a lot of silent errors can be avoided with this assert statement.


[Convert TFDS to HFDS] Extend script to also allow just converting a single file

Adds another argument to be able to convert only a single file


Name json file "squad.json" instead of "squad.py.json"


[Clean up] Datasets

Clean up `nlp/datasets` folder.

As I understood, eventually the `nlp/datasets` shall not exist anymore at all.

The folder `nlp/datasets/nlp` is kept for the moment, but won't be needed in the future, since it will live on S3 (actually it already does) at: `https://s3.console.aws.amazon.com/s3/buckets/datasets.huggingface.co/nlp/?region=us-east-1` and the different `dataset downloader scripts will be added to `nlp/src/nlp` when downloaded by the user.

The folder `nlp/datasets/checksums` is kept for now, but won't be needed anymore in the future.

The remaining folders/ files are leftovers from tensorflow-datasets and are not needed. The can be looked up in the private tensorflow-dataset repo.


Fix issue 6: error when the citation is missing in the DatasetInfo


Fix issue 5: allow empty datasets


Error when citation is not given in the DatasetInfo

Yes looks good to me.
Note that we may refactor quite strongly the `info.py` to make it a lot simpler (it's very complicated for basically a dictionary of info I think)

No, problem ^^ It might just be a temporary fix :)

The following error is raised when the `citation` parameter is missing when we instantiate a `DatasetInfo`:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jplu/dev/jplu/datasets/src/nlp/info.py", line 338, in __repr__
    citation_pprint = _indent('"""{}"""'.format(self.citation.strip()))
AttributeError: 'NoneType' object has no attribute 'strip'
```

I propose to do the following change in the `info.py` file. The method:
```python
def __repr__(self):
        splits_pprint = _indent("\n".join(["{"] + [
                "    '{}': {},".format(k, split.num_examples)
                for k, split in sorted(self.splits.items())
        ] + ["}"]))
        features_pprint = _indent(repr(self.features))
        citation_pprint = _indent('"""{}"""'.format(self.citation.strip()))
        return INFO_STR.format(
                name=self.name,
                version=self.version,
                description=self.description,
                total_num_examples=self.splits.total_num_examples,
                features=features_pprint,
                splits=splits_pprint,
                citation=citation_pprint,
                homepage=self.homepage,
                supervised_keys=self.supervised_keys,
                # Proto add a \n that we strip.
                license=str(self.license).strip())
```
Becomes:
```python
def __repr__(self):
        splits_pprint = _indent("\n".join(["{"] + [
                "    '{}': {},".format(k, split.num_examples)
                for k, split in sorted(self.splits.items())
        ] + ["}"]))
        features_pprint = _indent(repr(self.features))
        ## the strip is done only is the citation is given
        citation_pprint = self.citation

        if self.citation:
            citation_pprint = _indent('"""{}"""'.format(self.citation.strip()))
        return INFO_STR.format(
                name=self.name,
                version=self.version,
                description=self.description,
                total_num_examples=self.splits.total_num_examples,
                features=features_pprint,
                splits=splits_pprint,
                citation=citation_pprint,
                homepage=self.homepage,
                supervised_keys=self.supervised_keys,
                # Proto add a \n that we strip.
                license=str(self.license).strip())
```
And now it is ok. @thomwolf are you ok with this fix?


ValueError when a split is empty

To fix this I propose to modify only the file `arrow_reader.py` with few updates. First update, the following method:
```python
def _make_file_instructions_from_absolutes(
        name,
        name2len,
        absolute_instructions,
):
    """Returns the files instructions from the absolute instructions list."""
    # For each split, return the files instruction (skip/take)
    file_instructions = []
    num_examples = 0
    for abs_instr in absolute_instructions:
        length = name2len[abs_instr.splitname]
        if not length:
            raise ValueError(
                    'Split empty. This might means that dataset hasn\'t been generated '
                    'yet and info not restored from GCS, or that legacy dataset is used.')
        filename = filename_for_dataset_split(
                dataset_name=name,
                split=abs_instr.splitname,
                filetype_suffix='arrow')
        from_ = 0 if abs_instr.from_ is None else abs_instr.from_
        to = length if abs_instr.to is None else abs_instr.to
        num_examples += to - from_
        single_file_instructions = [{"filename": filename, "skip": from_, "take": to - from_}]
        file_instructions.extend(single_file_instructions)
    return FileInstructions(
            num_examples=num_examples,
            file_instructions=file_instructions,
    )
```
Becomes:
```python
def _make_file_instructions_from_absolutes(
        name,
        name2len,
        absolute_instructions,
):
    """Returns the files instructions from the absolute instructions list."""
    # For each split, return the files instruction (skip/take)
    file_instructions = []
    num_examples = 0
    for abs_instr in absolute_instructions:
        length = name2len[abs_instr.splitname]
        ## Delete the if not length and the raise
        filename = filename_for_dataset_split(
                dataset_name=name,
                split=abs_instr.splitname,
                filetype_suffix='arrow')
        from_ = 0 if abs_instr.from_ is None else abs_instr.from_
        to = length if abs_instr.to is None else abs_instr.to
        num_examples += to - from_
        single_file_instructions = [{"filename": filename, "skip": from_, "take": to - from_}]
        file_instructions.extend(single_file_instructions)
    return FileInstructions(
            num_examples=num_examples,
            file_instructions=file_instructions,
    )
```

Second update the following method:
```python
def _read_files(files, info):
    """Returns Dataset for given file instructions.

    Args:
        files: List[dict(filename, skip, take)], the files information.
            The filenames contain the absolute path, not relative.
            skip/take indicates which example read in the file: `ds.slice(skip, take)`
    """
    pa_batches = []
    for f_dict in files:
        pa_table: pa.Table = _get_dataset_from_filename(f_dict)
        pa_batches.extend(pa_table.to_batches())
    pa_table = pa.Table.from_batches(pa_batches)
    ds = Dataset(arrow_table=pa_table, data_files=files, info=info)
    return ds
```
Becomes:
```python
def _read_files(files, info):
    """Returns Dataset for given file instructions.

    Args:
        files: List[dict(filename, skip, take)], the files information.
            The filenames contain the absolute path, not relative.
            skip/take indicates which example read in the file: `ds.slice(skip, take)`
    """
    pa_batches = []
    for f_dict in files:
        pa_table: pa.Table = _get_dataset_from_filename(f_dict)
        pa_batches.extend(pa_table.to_batches())
    ## we modify the table only if there are some batches
    if pa_batches:
        pa_table = pa.Table.from_batches(pa_batches)
    ds = Dataset(arrow_table=pa_table, data_files=files, info=info)
    return ds
```

With these two updates it works now. @thomwolf are you ok with this changes?

Yes sounds good to me!
Do you want to make a PR? or I can do it as well

When a split is empty either TEST, VALIDATION or TRAIN I get the following error:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jplu/dev/jplu/datasets/src/nlp/load.py", line 295, in load
    ds = dbuilder.as_dataset(**as_dataset_kwargs)
  File "/home/jplu/dev/jplu/datasets/src/nlp/builder.py", line 587, in as_dataset
    datasets = utils.map_nested(build_single_dataset, split, map_tuple=True)
  File "/home/jplu/dev/jplu/datasets/src/nlp/utils/py_utils.py", line 158, in map_nested
    for k, v in data_struct.items()
  File "/home/jplu/dev/jplu/datasets/src/nlp/utils/py_utils.py", line 158, in <dictcomp>
    for k, v in data_struct.items()
  File "/home/jplu/dev/jplu/datasets/src/nlp/utils/py_utils.py", line 172, in map_nested
    return function(data_struct)
  File "/home/jplu/dev/jplu/datasets/src/nlp/builder.py", line 601, in _build_single_dataset
    split=split,
  File "/home/jplu/dev/jplu/datasets/src/nlp/builder.py", line 625, in _as_dataset
    split_infos=self.info.splits.values(),
  File "/home/jplu/dev/jplu/datasets/src/nlp/arrow_reader.py", line 200, in read
    return py_utils.map_nested(_read_instruction_to_ds, instructions)
  File "/home/jplu/dev/jplu/datasets/src/nlp/utils/py_utils.py", line 172, in map_nested
    return function(data_struct)
  File "/home/jplu/dev/jplu/datasets/src/nlp/arrow_reader.py", line 191, in _read_instruction_to_ds
    file_instructions = make_file_instructions(name, split_infos, instruction)
  File "/home/jplu/dev/jplu/datasets/src/nlp/arrow_reader.py", line 104, in make_file_instructions
    absolute_instructions=absolute_instructions,
  File "/home/jplu/dev/jplu/datasets/src/nlp/arrow_reader.py", line 122, in _make_file_instructions_from_absolutes
    'Split empty. This might means that dataset hasn\'t been generated '
ValueError: Split empty. This might means that dataset hasn't been generated yet and info not restored from GCS, or that legacy dataset is used.
```

How to reproduce:
```python
import csv

import nlp

class Bbc(nlp.GeneratorBasedBuilder):
    VERSION = nlp.Version("1.0.0")

    def __init__(self, **config):
        self.train = config.pop("train", None)
        self.validation = config.pop("validation", None)
        super(Bbc, self).__init__(**config)

    def _info(self):
        return nlp.DatasetInfo(builder=self, description="bla", features=nlp.features.FeaturesDict({"id": nlp.int32, "text": nlp.string, "label": nlp.string}))

    def _split_generators(self, dl_manager):
        return [nlp.SplitGenerator(name=nlp.Split.TRAIN, gen_kwargs={"filepath": self.train}),
                nlp.SplitGenerator(name=nlp.Split.VALIDATION, gen_kwargs={"filepath": self.validation}),
                nlp.SplitGenerator(name=nlp.Split.TEST, gen_kwargs={"filepath": None})]

    def _generate_examples(self, filepath):
        if not filepath:
            return None, {}

        with open(filepath) as f:
            reader = csv.reader(f, delimiter=',', quotechar="\"")
            lines = list(reader)[1:]

            for idx, line in enumerate(lines):
                yield idx, {"id": idx, "text": line[1], "label": line[0]}
```

```python
import nlp
dataset = nlp.load("bbc", builder_kwargs={"train": "bbc/data/train.csv", "validation": "bbc/data/test.csv"})
```


[Feature] Keep the list of labels of a dataset as metadata

Yes! I see mostly two options for this:
- a `Feature` approach like currently (but we might deprecate features)
- wrapping in a smart way the Dictionary arrays of Arrow: https://arrow.apache.org/docs/python/data.html?highlight=dictionary%20encode#dictionary-arrays

I would have a preference for the second bullet point.

This should be accessible now as a feature in dataset.info.features (and even have the mapping methods).

Hi,
I hope we could get a better documentation.
It took me more than 1 hour to found this way to get the label information.

Yes we are working on the doc right now, should be in the next release quite soon.

It would be useful to keep the list of the labels of a dataset as metadata. Either directly in the `DatasetInfo` or in the Arrow metadata.


[Feature] More dataset outputs

Yes!
- pandas will be a one-liner in `arrow_dataset`: https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.to_pandas
- for Spark I have no idea. let's investigate that at some point

For Spark it looks to be pretty straightforward as well https://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html but looks to be having a dependency to Spark is necessary, then nevermind we can skip it

Add the following dataset outputs:

- Spark
- Pandas


Issue to read a local dataset

My first bug report ❤
Looking into this right now!

Ok, there are some news, most good than bad :laughing:

The dataset script now became:
```python
import csv

import nlp

class Bbc(nlp.GeneratorBasedBuilder):
    VERSION = nlp.Version("1.0.0")

    def __init__(self, **config):
        self.train = config.pop("train", None)
        self.validation = config.pop("validation", None)
        super(Bbc, self).__init__(**config)

    def _info(self):
        return nlp.DatasetInfo(builder=self, description="bla", features=nlp.features.FeaturesDict({"id": nlp.int32, "text": nlp.string, "label": nlp.string}))

    def _split_generators(self, dl_manager):
        return [nlp.SplitGenerator(name=nlp.Split.TRAIN, gen_kwargs={"filepath": self.train}),
                nlp.SplitGenerator(name=nlp.Split.VALIDATION, gen_kwargs={"filepath": self.validation})]

    def _generate_examples(self, filepath):
        with open(filepath) as f:
            reader = csv.reader(f, delimiter=',', quotechar="\"")
            lines = list(reader)[1:]

            for idx, line in enumerate(lines):
                yield idx, {"id": idx, "text": line[1], "label": line[0]}

```

And the dataset folder becomes:
```
.
├── bbc
│   ├── bbc.py
│   └── data
│       ├── test.csv
│       └── train.csv
```
I can load the dataset by using the keywords arguments like this:
```python
import nlp
dataset = nlp.load("bbc", builder_kwargs={"train": "bbc/data/train.csv", "validation": "bbc/data/test.csv"})
```

That was the good part ^^ Because it took me some time to understand that the script itself is put in cache in `datasets/src/nlp/datasets/some-hash/bbc.py` which is very difficult to discover without checking the source code. It means that doesn't matter the changes you do to your original script it is taken into account. I think instead of doing a hash on the name (I suppose it is the name), a hash on the content of the script itself should be a better solution.

Then by diving a bit in the code I found the `force_reload` parameter [here](https://github.com/huggingface/datasets/blob/master/src/nlp/load.py#L50) but the call of this `load_dataset` method is done with the `builder_kwargs` as seen [here](https://github.com/huggingface/datasets/blob/master/src/nlp/load.py#L166) which is ok until the call to the builder is done as the builder do not have this `force_reload` parameter. To show as example, the previous load becomes:
```python
import nlp
dataset = nlp.load("bbc", builder_kwargs={"train": "bbc/data/train.csv", "validation": "bbc/data/test.csv", "force_reload": True})
```
Raises
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jplu/dev/jplu/datasets/src/nlp/load.py", line 283, in load
    dbuilder: DatasetBuilder = builder(path, name, data_dir=data_dir, **builder_kwargs)
  File "/home/jplu/dev/jplu/datasets/src/nlp/load.py", line 170, in builder
    builder_instance = builder_cls(**builder_kwargs)
  File "/home/jplu/dev/jplu/datasets/src/nlp/datasets/84d638d2a8ca919d1021a554e741766f50679dc6553d5a0612b6094311babd39/bbc.py", line 12, in __init__
    super(Bbc, self).__init__(**config)
TypeError: __init__() got an unexpected keyword argument 'force_reload'
```
So yes the cache is refreshed with the new script but then raises this error.

Ok great, so as discussed today, let's:
- have a main dataset directory inside the lib with sub-directories hashed by the content of the file
- keep a cache for downloading the scripts from S3 for now
- later: add methods to list and clean the local versions of the datasets (and the distant versions on S3 as well)

Side question: do you often use `builder_kwargs` for other things than supplying file paths? I was thinking about having a more easy to read and remember `data_files` argument maybe.

Good plan!

Yes I do use `builder_kwargs` for other things such as:
- dataset name
- properties to know how to properly read a CSV file: do I have to skip the first line in a CSV, which delimiter is used, and the columns ids to use.
- properties to know how to properly read a JSON file: which properties in a JSON object to read

Hello,

As proposed by @thomwolf, I open an issue to explain what I'm trying to do without success. What I want to do is to create and load a local dataset, the script I have done is the following:
```python
import os
import csv

import nlp

class BbcConfig(nlp.BuilderConfig):
    def __init__(self, **kwargs):
        super(BbcConfig, self).__init__(**kwargs)

class Bbc(nlp.GeneratorBasedBuilder):
    _DIR = "./data"
    _DEV_FILE = "test.csv"
    _TRAINING_FILE = "train.csv"

    BUILDER_CONFIGS = [BbcConfig(name="bbc", version=nlp.Version("1.0.0"))]

    def _info(self):
        return nlp.DatasetInfo(builder=self, features=nlp.features.FeaturesDict({"id": nlp.string, "text": nlp.string, "label": nlp.string}))

    def _split_generators(self, dl_manager):
        files = {"train": os.path.join(self._DIR, self._TRAINING_FILE), "dev": os.path.join(self._DIR, self._DEV_FILE)}

        return [nlp.SplitGenerator(name=nlp.Split.TRAIN, gen_kwargs={"filepath": files["train"]}),
                nlp.SplitGenerator(name=nlp.Split.VALIDATION, gen_kwargs={"filepath": files["dev"]})]

    def _generate_examples(self, filepath):
        with open(filepath) as f:
            reader = csv.reader(f, delimiter=',', quotechar="\"")
            lines = list(reader)[1:]

            for idx, line in enumerate(lines):
                yield idx, {"idx": idx, "text": line[1], "label": line[0]}

```

The dataset is attached to this issue as well:
[data.zip](https://github.com/huggingface/datasets/files/4476928/data.zip)

Now the steps to reproduce what I would like to do:
1. unzip data locally (I know the nlp lib can detect and extract archives but I want to reduce and facilitate the reproduction as much as possible)
2. create the `bbc.py` script as above at the same location than the unziped `data` folder.

Now I try to load the dataset in three different ways and none works, the first one with the name of the dataset like I would do with TFDS:
```python
import nlp
from bbc import Bbc
dataset = nlp.load("bbc")
```

I get:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 280, in load
    dbuilder: DatasetBuilder = builder(path, name, data_dir=data_dir, **builder_kwargs)
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 166, in builder
    builder_cls = load_dataset(path, name=name, **builder_kwargs)
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 88, in load_dataset
    local_files_only=local_files_only,
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/utils/file_utils.py", line 214, in cached_path
    if not is_zipfile(output_path) and not tarfile.is_tarfile(output_path):
  File "/opt/anaconda3/envs/transformers/lib/python3.7/zipfile.py", line 203, in is_zipfile
    with open(filename, "rb") as fp:
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

But @thomwolf told me that no need to import the script, just put the path of it, then I tried three different way to do:
```python
import nlp
dataset = nlp.load("bbc.py")
```
And
```python
import nlp
dataset = nlp.load("./bbc.py")
```
And
```python
import nlp
dataset = nlp.load("/absolute/path/to/bbc.py")
```

These three ways gives me:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 280, in load
    dbuilder: DatasetBuilder = builder(path, name, data_dir=data_dir, **builder_kwargs)
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 166, in builder
    builder_cls = load_dataset(path, name=name, **builder_kwargs)
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 124, in load_dataset
    dataset_module = importlib.import_module(module_path)
  File "/opt/anaconda3/envs/transformers/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'nlp.datasets.2fd72627d92c328b3e9c4a3bf7ec932c48083caca09230cebe4c618da6e93688.bbc'
```
Any idea of what I'm missing? or I might have spot a bug :)


changing nlp.bool to nlp.bool_