Diary of Working on Notas - The Issue with Memory

The Issue

Hello readers of 2coffee.dev, another week has passed, last Saturday Hanoi suddenly had a large thunderstorm, trees fell on the streets, roofs were torn off houses by nature's fury... but I was quite peaceful while in Nam Dinh countryside.

As a result, I had less time to work on Notas. But that doesn't mean there's nothing to tell. In fact, the story is even more interesting, perhaps only second to the week when I completed the synchronization function with Adapter.

As usual, now is the time to tell the story.

Interface for Mobile Devices

The mobile device interface has been built. Although it is not yet very refined, basic tasks on the small screen can be used. If you have heard of the "Mobile First" application standard - meaning prioritizing the interface for mobile devices (Mobile), then I am going... the opposite way.

Actually, Notas was originally designed to be used mainly on large screens (PC), as I wanted it to run smoothly on PC first and then work on the small screen. I spend more time in front of the computer than on the phone, and I use it more often, so prioritizing is not strange.

In general, transitioning the interface from PC to Mobile is not too difficult. Especially when Tailwind supports responsive through classes very well, all you need to do is add a screen prefix before, then to the CSS property, and that's it.

<div class="hidden lg:block">
...  
</div>

As in the example above, on the small screen, the div tag will be hidden, but when on the large screen, it will appear. It's simple, isn't it!

Being one of those who care about user experience, having many features is not necessarily a good thing, features need to be easy to use, convenient in operation is the top priority. Initially, I tried to keep all UI/UX from the PC version down to Mobile, but then realized it was not friendly at all. It's best to adjust it to suit mobile users.

For example, on PC, the menu of the folder and notes is activated by right-clicking or clicking on the three dots (...) then on Mobile, that is not suitable, the screen is too small to display, not to mention the "contrary" experience. Therefore, I redesigned the menu with a more professional modal display.

This took quite a bit of time, because basically I'm not a professional Front-end developer. I have to admit that I admire those who create interfaces. I wonder how much time you guys have spent refining each pixel.

The Issue with Memory

Import/Export data is a very basic feature that any note application should have. Sometimes we want to back up notes for safety or serve a personal purpose, so I added this feature to Notas.

By the way, once the import/export is done, another idea suddenly flashes: Why don't I try to test Notas' performance? By importing a large number of notes to see how it goes!

After writing a piece of JavaScript code to generate a JSON file containing hundreds of notes with random lengths from 1000 to 2000 words, Notas imported it instantly and there was nothing to say. After increasing to a few thousand, an error suddenly appeared. Specifically, Local Storage reported... out of memory.

For those who don't know, I used Local Storage to store user data, and then through Adapter to synchronize data to the online server. However, Local Storage has a fairly low memory limit, only about a few MB. If you store more, you will encounter the above error.

Recognizing this early, I researched and learned that IndexedDB is another solution to replace Local Storage. IndexedDB has a much larger memory capacity (can be up to 80% of hard drive capacity) depending on the browser. IndexedDB is also supported by many modern browsers today. It is also a key-value database similar to Local Storage.

The localForage library provides an API similar to Local Storage for quick switching from Local Storage to IndexedDB. Because the syntax for reading/writing data is very similar, it doesn't take too much time to switch. All you need to do is replace the localStorage call functions with localForage and that's it.

Feeling triumphant, this time I decided to raise the number to 10,000 notes. And... boom! Only a few thousand notes appeared in Notas. Strange, where did the rest go? Is there an error?

After some fumbling, it turns out, the remaining notes are not lost, they just haven't been imported yet. Or in other words, Notas is still in the process of importing notes into IndexedDB. And if I remember correctly, it took more than 30 minutes for 10,000 notes to be fully imported into Notas. Wow! Why is it so slow?

It turns out the answer lies in IndexedDB. After a round of forums, there are many humorous stories revolving around IndexedDB. Why is it slow? Why doesn't the browser have a decent database? At least a familiar SQL type database? (Actually, there is, WebSQL is a database for browsers, but it is not standardized so many browsers are not interested, even removed from the browser).

To this point, we have to go back to the past a bit, before the time when the first lines of Notas code were written: I thought about choosing a database. For many programmers, SQL is a very familiar data query language, so if a browser or library supports SQL, there is nothing to say.

SQL.js is one of the few that can do this. It is converted from SQLite native to WebAssembly to run in the browser. With just a few simple lines of code, you can now use SQLite right in your browser. If so, isn't everything solved already?

Unfortunately not! SQL.js is only partially supported in the browser. It cannot read/write directly to a sqlite file because the browser's permissions do not allow it. Therefore, all data will be deleted when you close the session or refresh the web page. Haiz! I have to go back to LocalStorage, oh... IndexedDB.

Not giving up. I continued to search for a solution to store data on the browser. Another name appeared: RxDB.

RxDB is introduced as "The local Database for JavaScript Applications". That's right, it provides a solution to store data in the browser. And surprisingly, some of my Notas design ideas are quite similar to RxDB.

RxDB also uses an Interface layer to interact with the actual database behind it, called RxStorage. RxDB acts as an intermediary layer to store data down to the browser. For example, when you declare the database models as usual, call the add/edit/delete data functions of RxDB as usual. Then RxDB will have RxStorage to change data down to Local Storage, IndexedDB, WebSQL, or even in the computer's RAM.

In addition, it also provides an offline/online data synchronization solution, also known as Offline Support. The document clearly describes how to synchronize based on pull/push functions similar to the idea that Notas has implemented. It's just that they do it in more detail and complexity to meet most usage requirements.

However, many RxStorage need to be paid to be used. For example, if you want to use IndexedDB, you still have to pay!!! The amount of money spent is not small. Is it free? There is! But the application is not high. It seems I also learned a bit up here and then went out.

But it's not over yet. When one door closes, another opens. Specifically, a 2021 article A future for SQL on the web by a programmer. He was so frustrated with IndexedDB that he created a library to support SQL right in the browser.

By combining SQL.js and IndexedDB. James Long has utilized IndexedDB as a "hard drive" to store data for the SQL.js library - originally SQLite. That's right, it means you can now read/write data with SQL right in the browser, and of course, it's persistently stored even when the machine is turned off.

He also pointed out a very absurd thing is that although using IndexedDB for storage, the query speed of SQLite on this is still dozens of times faster than IndexedDB. Perhaps that's why the name "absurd" is given to the library.

So, Notas has hope to optimize performance in the near future. But that might be a priority task after being launched. I will complete Notas with IndexedDB first to be able to launch it to readers. It's rare to have someone create 10,000 notes in a short time.

Plan for Next Week

The next week will be the time to perfect the UI/UX on both PC and Mobile. Because many features are still incomplete, my task is to unify the usage flows. In addition, I will optimize the code, remove redundant code, or restructure.

And you, do you have any questions or concerns about the process of making Notas? Leave a comment to let me and everyone know!