Timeline for Approach for parsing and indexing very large files
Current License: CC BY-SA 3.0
15 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 26, 2020 at 17:30 | vote | accept | Matt | ||
| Mar 20, 2017 at 10:29 | history | edited | CommunityBot | replaced http://meta.stackexchange.com/ with https://meta.stackexchange.com/ | |
| Feb 8, 2017 at 4:22 | answer | added | Wyatt Barnett | timeline score: 2 | |
| Feb 7, 2017 at 19:01 | answer | added | Thomas Carlisle | timeline score: 1 | |
| Feb 7, 2017 at 4:38 | answer | added | John Wu | timeline score: 3 | |
| Feb 6, 2017 at 22:38 | answer | added | Joppe | timeline score: 2 | |
| Feb 5, 2017 at 12:59 | comment | added | Apalala | Please post an anonymized sample of the log file format and of a typical query? | |
| Jan 9, 2017 at 3:32 | history | tweeted | twitter.com/StackSoftEng/status/818299362681954304 | ||
| Dec 19, 2016 at 20:46 | comment | added | Blrfl | If you're doing full-text search, consider using a full-text indexer like Solr or Lucene, which underpins it. | |
| Dec 19, 2016 at 20:33 | history | edited | Matt | CC BY-SA 3.0 | added 656 characters in body |
| Dec 19, 2016 at 20:23 | comment | added | Christophe | Can you clarify why you need to feed 3 different databases ? Can you at which rythm the logs are parsed (every day ? every week ? every message ? | |
| Dec 19, 2016 at 11:01 | review | Close votes | |||
| Jan 10, 2017 at 3:02 | |||||
| Dec 19, 2016 at 8:32 | comment | added | Basile Starynkevitch | You should explain a bit more what these proprietary log files are, what is their format, how are they written (or grown)? Of course, the approach depends a lot on the operating system & file system involved. | |
| Dec 19, 2016 at 8:29 | comment | added | Basile Starynkevitch | What make you think that Sqlite cannot work on gigabyte-sized files? It could (at least if you create appropriate indexes). See this. And you might parse the log files incrementally. | |
| Dec 19, 2016 at 7:28 | history | asked | Matt | CC BY-SA 3.0 |