3

I am trying to open large files such as .log or .txt or even source files(range from 40 MB to 5 GB or even more) without loading everything into ram. Something what can read and edit large file byte by byte and represent them by scrolling for example up and down.

Best Solution would be software whit a build in function like that or say like a decent editor f.e. VS Code with an add on.

All I could find so far is software that loads everything into ram which is not only expensive but also costs time.

Improve this question
5
  • "cost time"? on my 8 year old computer 300MB file load in notepad++ in 2-3 seconds Commented Jul 10, 2021 at 16:00
  • 1
    What operating system and version are you using? It would, for example, be inappropriate to suggest Notepad++ if you're using MacOS. Commented Jul 10, 2021 at 18:18
  • Also, what's your budget? CRiSP (suggested in an answer below), at $275, is comparatively expensive, and might not be acceptable if you don't have that kind of funding available. Commented Jul 10, 2021 at 18:25
  • @JeffZeitlin I mainly use Windows and yes if possible a FOSS software (or something with a reasonable price) 275 seems really expensive. Commented Jul 11, 2021 at 11:34
  • Windows: Free for non-commercial use: BssEditor - no limit regarding file-size. Commented Jul 11, 2021 at 12:23

4 Answers 4

Reset to default
3

If it is only viewing and not editing such large files then you should take a look onto the open source Multi-Platform log viewer klogg. It is pretty fast, allows to filter the opened files and highlight certain elements.

Improve this answer
4
  • Thank you, Github confirm no loading into memory. So if there is reading large files is writing & reading difficult to archive? Or why is it not implemented? Commented Jul 11, 2021 at 11:30
  • 1
    @MIkey Writing is very difficult to achieve because files are usually written byte-by byte. Imagine you want to insert one character at the beginning of a 1GB file and save. On most OS inserting data is not available for files, only overwrite is possible. This means that for inserting just one byte you have to re-write every byte that is after the inserted position in the file. Commented Jul 11, 2021 at 11:47
  • Got it, thanks! U said "On most OS". Can i assume that there are some OS`s with a Filesystem that allows inserting data? Or is it bad practice or some other problem why Windows wont support inserting? Commented Jul 11, 2021 at 13:29
  • @MIkey Well at least I don't know any OS or file-system that allows insert operations. Any on the "why" I can only guess: because of complexity? File-systems usually operate on blocks (e.g. 4K). If you would allow inserting a singe byte then you would have to allocate a whole block for one byte and record that somehow. The more insertions you make the more complex the calculation gets where a block starts or ends which is important for random access within a file. Thus a file-system which allows insertion would get very ineffective (-> slow) regarding random-accesss speed. Commented Jul 11, 2021 at 13:54
1

If you are on windows and really need to edit moderately large files (<2 GB) then Notepad++ is a good option:

  • Free
  • Powerful
  • Reasonably simple to use

On Linux/Mac/Windows you can use vim but it will get slow on large files and may crash on very large (>10GB) files:

  • Free
  • Very Powerful
  • Not so simple to use especially if you use it infrequently

However, if you are truly needing to manipulate very large files you probably don't want to be spending weeks scrolling through them reading them you are more likely to be looking to extract specific information from them. This is where command line tools such as sed, awk & grep come into their own - they can take a little while to learn to use well but are fast and powerful. You could also use a programming language such as python of course. All of these are:

  • Free
  • Available for just about every platform (even Windows)
  • Insanely powerful
  • Fast an reliable
  • Can be scripted to as to get reproducible results
  • Work line by line, (or can be told to in Pythons case), so can process files that exceed RAM size, (even multi-TB files).
  • You need to learn how to use them effectively but there is a lot of often free training material available.
Improve this answer
3
  • 1
    You forgot one "predicate" of vim: It's usability for people not using it regularly is to say it nice "very special". Commented Jul 11, 2021 at 11:49
  • @Robert - Yes indeed - I have added a mention of infrequent use - <esc>:q! is the one key sequence to never forget! Commented Jul 11, 2021 at 11:53
  • 1
    Thank you for the detailed answer! I will look into some of these. Commented Jul 11, 2021 at 13:31
0

I have used the "crisp" text editor for large data files and have never found a file size limit either in line width or in number of lines. It has been around for a long time. It has a myriad of advanced editing modes including hex, macros, search/replace, and many others. It is commercial. Available on linux, windows, and mac.

I still use a 10 year old windows copy on my linux machine and run it through wine.

Have a look. tom kosvic

Improve this answer
1
  • Could you please add a link? Commented Jul 10, 2021 at 20:12
0

I was using a long time BareTail but was missing more and more features dealing with huge log files. So I write my own app loxx which should meet your requirements (opens huge files very fast, supports highlighting, fast search etc...) Have a try -- every feedback is welcome.
Loxx runs on Windows 32/64 systems 10/11 (7/8 should work but is not tested).

Improve this answer
1
  • What OS is this for? Commented Jun 3 at 10:27

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.