Skip to content

Commit 93ad023

Browse files
committed
Merge branch 'dev'
2 parents a8af67d + a77cbc0 commit 93ad023

13 files changed

+4688
-463
lines changed

README.md

Lines changed: 45 additions & 78 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
1-
![Laravel Simple CSV](https://cdn.rawgit.com/bayareawebpro/laravel-simple-csv/97d15ca6/screenshot.png "Laravel Simple CSV")
2-
31
# Simple CSV for Laravel
2+
43
[![Generic badge](https://img.shields.io/badge/Build-Passing-ok.svg)]()
5-
[![Generic badge](https://img.shields.io/badge/License-MIT-orange.svg)]()
6-
[![Generic badge](https://img.shields.io/badge/Version-1.0-blue.svg)]()
4+
[![Generic badge](https://img.shields.io/badge/License-MIT-green.svg)]()
5+
[![Generic badge](https://img.shields.io/badge/Version-2.0-blue.svg)]()
76

87
## Features
9-
- Import to Collection - Export from Collection.
10-
- Low(er) Memory Consumption by use of Generators.
8+
- Import to LazyCollection.
9+
- Export from Collection, LazyCollection, Iterable, Generator, Array.
10+
- Low(er) Memory Consumption by use of LazyCollection Generators.
1111
- Uses Native PHP SplFileObject.
1212
- Facade Included.
1313

@@ -17,7 +17,6 @@ Simply require the package and Laravel will Auto-Discover the Service Provider.
1717
composer require bayareawebpro/laravel-simple-csv
1818
```
1919

20-
2120
## Usage:
2221

2322
### Import to Collection
@@ -29,45 +28,62 @@ $collection = SimpleCsv::import(storage_path('table.csv'));
2928
```
3029
___
3130

32-
### Export from Collection to File
33-
1) Pass a collection to the export method, which returns the chained save method.
34-
2) Specify the path when you call the save method.
31+
### Import
32+
```
33+
$lazyCollection = SimpleCsv::import(storage_path('collection.csv'));
34+
```
35+
36+
### Export to File
3537
```
3638
use SimpleCsv;
37-
use DB;
3839
39-
$collection = DB::table('users')->get(['id', 'name', 'email']);
40-
$exporter = SimpleCsv::export($collection);
41-
$exporter->save(storage_path('table.csv'));
40+
// Collection
41+
SimpleCsv::export(
42+
Collection::make(...),
43+
storage_path('collection.csv')
44+
);
45+
46+
// LazyCollection
47+
SimpleCsv::export(
48+
LazyCollection::make(...),
49+
storage_path('collection.csv')
50+
);
51+
52+
// Generator (Cursor)
53+
SimpleCsv::export(
54+
User::query()->where(...)->limit(500)->cursor(),
55+
storage_path('collection.csv')
56+
);
57+
58+
// Array
59+
SimpleCsv::export(
60+
[...],
61+
storage_path('collection.csv')
62+
);
4263
```
43-
### Export from Collection to Download Stream
44-
1) Pass a collection to the export method, which returns the chained download response streamer.
45-
2) Specify the filename when you call the download method.
46-
3) Return the response provided by the download method.
64+
65+
### Export Download Stream
4766
```
4867
use SimpleCsv;
49-
use DB;
50-
5168
public function download(Request $request)
5269
{
53-
$collection = DB::table('users')->get(['id', 'name', 'email']);
54-
$exporter = SimpleCsv::export($collection);
55-
return $exporter->download('table.csv');
70+
$collection = LazyCollection::make(...);
71+
return SimpleCsv::download($collection, 'download.csv');
5672
}
5773
```
5874

5975
#### Extended Options
6076
```
61-
SimpleCsv::import($path = '/some/file.csv', $callback = function, $chunk = 500, $delimiter = ",", $enclosure = "\"", $escape = "\\");
62-
SimpleCsv::export($collection, $delimiter = ",", $enclosure = "\"", $escape = "\\");
77+
SimpleCsv::make($delimter, $enclosure, $escape)->export(...);
78+
SimpleCsv::make($delimter, $enclosure, $escape)->import(...);
6379
```
6480

6581
## File Splitting Utility
6682
A file splitting utility has been included that will break large CSV files into chunks
6783
(while retaining column headers) which you can move/delete after importing.
6884
This can help with automating the import of large data sets.
6985

70-
Tip: Find your Bash Shell Binary Path: ``$ which sh``
86+
Tip: Find your Bash Shell Binary Path: `which sh`
7187

7288
```
7389
/bin/sh vendor/bayareawebpro/laravel-simple-csv/split-csv.sh /Projects/laravel/storage/big-file.csv 5000
@@ -80,56 +96,7 @@ etc...
8096
```
8197

8298
## Speed Tips
83-
- Queries are faster when you specify fields in the `get()` or `select()` method.
84-
- Using the DB Facade instead of Eloquent can yield faster results.
99+
- Using Lazy Collections is the preferred method.
85100
- Using the queue worker, you can import a several thousand rows at a time without much impact.
86-
- Be sure to use "Database Transactions", "Chunking" and "Timeout Detection" to insure safe imports.
87-
- [Article: How to Insert & Update Many at Once](https://medium.com/@danielalvidrez/laravel-query-builder-macros-fe176d34135e)
88-
89-
### Chunk Export:
90-
Chunk Query into multiple 10,000 row files and get the download paths.
91-
```
92-
$paths = [];
93-
$query = DB::table('users')->select(['id', 'name', 'email'])->orderBy('id');
94-
$query->chunk(10000, function($chunk, $index){
95-
$path = storage_path("chunk-{$index}.csv");
96-
SimpleCsv::export($chunk)->save($path);
97-
array_push($paths, $path)
98-
});
99-
100-
File Output:
101-
chunk-1.csv
102-
chunk-2.csv
103-
chunk-3.csv
104-
...
105-
```
106-
107-
### Chunk Import
108-
Import file into the database, but group the insert queries as chunks.
109-
```
110-
DB::transaction(function(){
111-
$chunks = SimpleCsv::import(storage_path('table.csv'))->chunk(1000);
112-
$chunks->each(function($chunk){
113-
DB::insert($chunk->toArray());
114-
});
115-
});
116-
```
117-
118-
Expose chunk callback to generator allowing lower memory consumption for large files:
119-
```
120-
DB::transaction(function(){
121-
SimpleCsv::import(storage_path('table.csv'), function($chunk){
122-
DB::insert($chunk->toArray());
123-
}, 1000);
124-
});
125-
```
126-
127-
#### DebugBar Timeline Results:
128-
129-
| Operation | Chunk | Total Entries | Total Memory | Framework Memory | Actual Memory | Total Time | Framework Time | Actual Time |
130-
| :-------: | :----: | :-----------: | :----------: | :--------------: | :-----------: | :--------: | :------------: | :---------: |
131-
| Import | 10,000 | 10,000 | 15.2 | 8.09 | 7.11 | 0.311 | 0.166 | 0.145 |
132-
| Export | 10,000 | 10,000 | 18.3 | 8.09 | 10.21 | 0.484 | 0.166 | 0.318 |
133-
| Export | 10,000 | 221,443 | 18.09 | 8.09 | 10 | 11.21 | 0.166 | 11.044 |
134-
135-
Test Machine: *MacPro (3.1) 2.8Ghz (Dual) Quad Core / 18GB 800Mhz FB-DIM Memory / SSD*
101+
- Be sure to use "Database Transactions" and "Timeout Detection" to insure safe imports.
102+
- [Article: How to Insert & Update Many at Once](https://medium.com/@danielalvidrez/laravel-query-builder-macros-fe176d34135e)

composer.json

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,25 @@
1010
],
1111
"require": {
1212
"php": "^7.1.3",
13-
"laravel/framework": "5.6.*|5.7.*|5.8.*|6.*"
13+
"illuminate/support": "6.*"
14+
},
15+
"require-dev": {
16+
"orchestra/testbench": "^4.0",
17+
"phpunit/phpunit": "^8.0"
1418
},
1519
"autoload": {
1620
"psr-4": {
1721
"BayAreaWebPro\\SimpleCsv\\": "src/"
1822
}
1923
},
24+
"autoload-dev": {
25+
"psr-4": {
26+
"BayAreaWebPro\\SimpleCsv\\Tests\\": "tests"
27+
}
28+
},
29+
"scripts": {
30+
"test": "vendor/bin/phpunit"
31+
},
2032
"extra": {
2133
"laravel": {
2234
"providers": [

0 commit comments

Comments
 (0)