Skip to content

Commit 0b18c36

Browse files
bayareawebprobayareawebpro
committed
updated readme
1 parent e38f17f commit 0b18c36

File tree

3 files changed

+80
-126
lines changed

3 files changed

+80
-126
lines changed

‎README.md‎

Lines changed: 45 additions & 78 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
1-
![Laravel Simple CSV](https://cdn.rawgit.com/bayareawebpro/laravel-simple-csv/97d15ca6/screenshot.png "Laravel Simple CSV")
2-
31
# Simple CSV for Laravel
2+
43
[![Generic badge](https://img.shields.io/badge/Build-Passing-ok.svg)]()
5-
[![Generic badge](https://img.shields.io/badge/License-MIT-orange.svg)]()
6-
[![Generic badge](https://img.shields.io/badge/Version-1.0-blue.svg)]()
4+
[![Generic badge](https://img.shields.io/badge/License-MIT-green.svg)]()
5+
[![Generic badge](https://img.shields.io/badge/Version-2.0-blue.svg)]()
76

87
## Features
9-
- Import to Collection - Export from Collection.
10-
- Low(er) Memory Consumption by use of Generators.
8+
- Import to LazyCollection.
9+
- Export from Collection, LazyCollection, Iterable, Generator, Array.
10+
- Low(er) Memory Consumption by use of LazyCollection Generators.
1111
- Uses Native PHP SplFileObject.
1212
- Facade Included.
1313

@@ -17,7 +17,6 @@ Simply require the package and Laravel will Auto-Discover the Service Provider.
1717
composer require bayareawebpro/laravel-simple-csv
1818
```
1919

20-
2120
## Usage:
2221

2322
### Import to Collection
@@ -29,45 +28,62 @@ $collection = SimpleCsv::import(storage_path('table.csv'));
2928
```
3029
___
3130

32-
### Export from Collection to File
33-
1) Pass a collection to the export method, which returns the chained save method.
34-
2) Specify the path when you call the save method.
31+
### Import
32+
```
33+
$lazyCollection = SimpleCsv::import(storage_path('collection.csv'));
34+
```
35+
36+
### Export to File
3537
```
3638
use SimpleCsv;
37-
use DB;
3839
39-
$collection = DB::table('users')->get(['id', 'name', 'email']);
40-
$exporter = SimpleCsv::export($collection);
41-
$exporter->save(storage_path('table.csv'));
40+
// Collection
41+
SimpleCsv::export(
42+
Collection::make(...),
43+
storage_path('collection.csv')
44+
);
45+
46+
// LazyCollection
47+
SimpleCsv::export(
48+
LazyCollection::make(...),
49+
storage_path('collection.csv')
50+
);
51+
52+
// Generator (Cursor)
53+
SimpleCsv::export(
54+
User::query()->where(...)->limit(500)->cursor(),
55+
storage_path('collection.csv')
56+
);
57+
58+
// Array
59+
SimpleCsv::export(
60+
[...],
61+
storage_path('collection.csv')
62+
);
4263
```
43-
### Export from Collection to Download Stream
44-
1) Pass a collection to the export method, which returns the chained download response streamer.
45-
2) Specify the filename when you call the download method.
46-
3) Return the response provided by the download method.
64+
65+
### Export Download Stream
4766
```
4867
use SimpleCsv;
49-
use DB;
50-
5168
public function download(Request $request)
5269
{
53-
$collection = DB::table('users')->get(['id', 'name', 'email']);
54-
$exporter = SimpleCsv::export($collection);
55-
return $exporter->download('table.csv');
70+
$collection = LazyCollection::make(...);
71+
return SimpleCsv::download($collection, 'download.csv');
5672
}
5773
```
5874

5975
#### Extended Options
6076
```
61-
SimpleCsv::import($path = '/some/file.csv', $callback = function, $chunk = 500, $delimiter = ",", $enclosure = "\"", $escape = "\\");
62-
SimpleCsv::export($collection, $delimiter = ",", $enclosure = "\"", $escape = "\\");
77+
SimpleCsv::make($delimter, $enclosure, $escape)->export(...);
78+
SimpleCsv::make($delimter, $enclosure, $escape)->import(...);
6379
```
6480

6581
## File Splitting Utility
6682
A file splitting utility has been included that will break large CSV files into chunks
6783
(while retaining column headers) which you can move/delete after importing.
6884
This can help with automating the import of large data sets.
6985

70-
Tip: Find your Bash Shell Binary Path: ``$ which sh``
86+
Tip: Find your Bash Shell Binary Path: `which sh`
7187

7288
```
7389
/bin/sh vendor/bayareawebpro/laravel-simple-csv/split-csv.sh /Projects/laravel/storage/big-file.csv 5000
@@ -80,56 +96,7 @@ etc...
8096
```
8197

8298
## Speed Tips
83-
- Queries are faster when you specify fields in the `get()` or `select()` method.
84-
- Using the DB Facade instead of Eloquent can yield faster results.
99+
- Using Lazy Collections is the preferred method.
85100
- Using the queue worker, you can import a several thousand rows at a time without much impact.
86-
- Be sure to use "Database Transactions", "Chunking" and "Timeout Detection" to insure safe imports.
87-
- [Article: How to Insert & Update Many at Once](https://medium.com/@danielalvidrez/laravel-query-builder-macros-fe176d34135e)
88-
89-
### Chunk Export:
90-
Chunk Query into multiple 10,000 row files and get the download paths.
91-
```
92-
$paths = [];
93-
$query = DB::table('users')->select(['id', 'name', 'email'])->orderBy('id');
94-
$query->chunk(10000, function($chunk, $index){
95-
$path = storage_path("chunk-{$index}.csv");
96-
SimpleCsv::export($chunk)->save($path);
97-
array_push($paths, $path)
98-
});
99-
100-
File Output:
101-
chunk-1.csv
102-
chunk-2.csv
103-
chunk-3.csv
104-
...
105-
```
106-
107-
### Chunk Import
108-
Import file into the database, but group the insert queries as chunks.
109-
```
110-
DB::transaction(function(){
111-
$chunks = SimpleCsv::import(storage_path('table.csv'))->chunk(1000);
112-
$chunks->each(function($chunk){
113-
DB::insert($chunk->toArray());
114-
});
115-
});
116-
```
117-
118-
Expose chunk callback to generator allowing lower memory consumption for large files:
119-
```
120-
DB::transaction(function(){
121-
SimpleCsv::import(storage_path('table.csv'), function($chunk){
122-
DB::insert($chunk->toArray());
123-
}, 1000);
124-
});
125-
```
126-
127-
#### DebugBar Timeline Results:
128-
129-
| Operation | Chunk | Total Entries | Total Memory | Framework Memory | Actual Memory | Total Time | Framework Time | Actual Time |
130-
| :-------: | :----: | :-----------: | :----------: | :--------------: | :-----------: | :--------: | :------------: | :---------: |
131-
| Import | 10,000 | 10,000 | 15.2 | 8.09 | 7.11 | 0.311 | 0.166 | 0.145 |
132-
| Export | 10,000 | 10,000 | 18.3 | 8.09 | 10.21 | 0.484 | 0.166 | 0.318 |
133-
| Export | 10,000 | 221,443 | 18.09 | 8.09 | 10 | 11.21 | 0.166 | 11.044 |
134-
135-
Test Machine: *MacPro (3.1) 2.8Ghz (Dual) Quad Core / 18GB 800Mhz FB-DIM Memory / SSD*
101+
- Be sure to use "Database Transactions" and "Timeout Detection" to insure safe imports.
102+
- [Article: How to Insert & Update Many at Once](https://medium.com/@danielalvidrez/laravel-query-builder-macros-fe176d34135e)

‎src/SimpleCsv.php‎

Lines changed: 34 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,10 @@
22

33
namespace BayAreaWebPro\SimpleCsv;
44

5+
use Iterator;
6+
use Generator;
57
use SplFileObject;
8+
use Illuminate\Support\Collection;
69
use Illuminate\Support\LazyCollection;
710

811
class SimpleCsv
@@ -28,44 +31,21 @@ class SimpleCsv
2831

2932
public function __construct()
3033
{
31-
$this->escape = static::ESCAPE;
32-
$this->delimiter = static::DELIMITER;
33-
$this->enclosure = static::ENCLOSURE;
34+
$this->escape = self::ESCAPE;
35+
$this->delimiter = self::DELIMITER;
36+
$this->enclosure = self::ENCLOSURE;
3437
$this->headers = null;
3538
$this->file = null;
3639
}
3740

38-
/**
39-
* Set Delimiter String
40-
* @param string $delimiter
41-
* @return $this
42-
*/
43-
public function setDelimiter(string $delimiter): self
44-
{
41+
public function make(
42+
string $delimiter = self::DELIMITER,
43+
string $enclosure = self::ENCLOSURE,
44+
string $escape = self::ESCAPE
45+
){
4546
$this->delimiter = $delimiter;
46-
return $this;
47-
}
48-
49-
/**
50-
* Set Enclosure String
51-
* @param string $enclosure
52-
* @return $this
53-
*/
54-
public function setEnclosure(string $enclosure): self
55-
{
5647
$this->enclosure = $enclosure;
57-
return $this;
58-
}
59-
60-
/**
61-
* Set Escape String
62-
* @param string $escape
63-
* @return $this
64-
*/
65-
public function setEscape(string $escape): self
66-
{
6748
$this->escape = $escape;
68-
return $this;
6949
}
7050

7151
/**
@@ -99,12 +79,12 @@ protected function isInValidLine($line): bool
9979

10080
/**
10181
* Export to FileSystem.
102-
* @param LazyCollection $collection
82+
* @param LazyCollection|Collection|array $collection
10383
* @param string $path
10484
* @return self
10585
* @throws \Throwable
10686
*/
107-
public function export(LazyCollection $collection, string $path): self
87+
public function export($collection, string $path): self
10888
{
10989
//Get the file object.
11090
if (!file_exists($path)) touch($path);
@@ -116,12 +96,12 @@ public function export(LazyCollection $collection, string $path): self
11696

11797
/**
11898
* Download Response
119-
* @param LazyCollection $collection
99+
* @param LazyCollection|Collection|array $collection
120100
* @param $filename string
121101
* @return \Symfony\Component\HttpFoundation\StreamedResponse
122102
* @throws \Throwable
123103
*/
124-
public function download(LazyCollection $collection, string $filename)
104+
public function download($collection, string $filename)
125105
{
126106
return response()->streamDownload(function () use ($collection) {
127107
$this->openFileObject('php://output', 'w');
@@ -181,19 +161,26 @@ protected function closeFileObject(): void
181161

182162
/**
183163
* @param LazyCollection $collection
164+
* @throws \Throwable
184165
*/
185-
protected function writeLines(LazyCollection $collection): void
166+
protected function writeLines($collection): void
186167
{
187-
$collection->chunk(500)->each(function (LazyCollection $chunk) {
188-
$chunk->each(function ($entry) {
189-
if (!$this->headers) {
190-
$this->headers = array_keys($this->flattenRow($entry));
191-
$this->writeLine($this->headers);
192-
}
193-
$this->writeLine(array_values($this->flattenRow($entry)));
194-
unset($entry);
195-
});
196-
unset($chunk);
197-
});
168+
if(
169+
!$collection instanceof Iterator &&
170+
!$collection instanceof Generator &&
171+
!$collection instanceof Collection &&
172+
!$collection instanceof LazyCollection &&
173+
!is_array($collection)
174+
){
175+
throw new \Exception("Non-Iterable Object cannot be iterated.");
176+
}
177+
foreach ($collection as $entry){
178+
if (!$this->headers) {
179+
$this->headers = array_keys($this->flattenRow($entry));
180+
$this->writeLine($this->headers);
181+
}
182+
$this->writeLine(array_values($this->flattenRow($entry)));
183+
unset($entry);
184+
}
198185
}
199186
}

‎tests/Unit/DefaultTest.php‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
namespace BayAreaWebPro\SimpleCsv\Tests\Unit;
44

55

6+
use Illuminate\Database\Eloquent\Model;
67
use Illuminate\Support\Str;
78
use Illuminate\Support\Facades\File;
89
use Illuminate\Support\LazyCollection;
@@ -82,7 +83,6 @@ public function test_export_files_and_restore()
8283

8384
public function test_can_download_streams()
8485
{
85-
8686
$items = $this->getCollectionData()->toArray();
8787

8888
$collectionLazy = LazyCollection::make($items);

0 commit comments

Comments
 (0)