Opensource.com https://opensource.com/ en What's new with Awk? https://opensource.com/article/22/10/whats-new-awk <span class="field field--name-title field--type-string field--label-hidden">What&#039;s new with Awk?</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><a title="View user profile." href="/users/jim-hall" class="username">Jim Hall</a></span> <span class="field field--name-created field--type-created field--label-hidden">Sat, 10/01/2022 - 03:00</span> <div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>Awk is a <a href="https://opensource.com/resources/what-awk" target="_blank">powerful scripting tool</a> that makes it easy to process text. Awk scripts use a <i>pattern-action</i> syntax, where Awk performs an action for every line in a file that matches a pattern. This provides a flexible yet powerful scripting language to deal with text. For example, the one-line Awk script <code>/error/ {print $1, $2, $3}</code> will print the first three space-delimited fields for any line that contains the word <code>error</code>.</p> <p>While we also have the GNU variant of Awk, called Gawk, the original Awk remains under development. Recently, Brian Kernighan started a project to add Unicode support to Awk. I met with Brian to ask about the origins of Awk and his recent development work on Awk.</p> <p><b>Jim Hall: Awk is a great tool to parse and process text. How did it start?</b></p> <p><b>Brian Kernighan:</b> The most direct influence was a tool that Marc Rochkind developed while working on the Programmer's Workbench system at Bell Labs. As I remember it now, Marc's program took a list of regular expressions and created a C program that would read an input file. Whenever the program found a match for one of the regular expressions, it printed the matching line. It was designed for creating error checking to run over log files from telephone operations data. It was such a neat idea—Awk is just a generalization.</p> <p><b>Jim: AWK stands for the three of you who created it: Al Aho, Peter Weinberger, and Brian Kernighan. How did the three of you design and create Awk?</b></p> <p><b>Brian:</b> Al was interested in regular expressions and had recently implemented egrep, which provided a very efficient lazy-evaluation technique for a much bigger class of regular expressions than what grep provided. That gave us a syntax and working code.</p> <p>Peter had been interested in databases, and as part of that he had some interest in report generation, like the RPG language that IBM provided. And I had been trying to figure out some kind of editing system that made it possible to handle strings and numbers with more or less equal ease.</p> <p>We explored designs, but not for a long time. I think Al may have provided the basic pattern-action paradigm, but that was implicit in a variety of existing tools, like grep, the stream editor sed, and in the language tools YACC and Lex that we used for implementation. Naturally, the action language had to be C-like.</p> <p><b>Jim: How was Awk first used at Bell Labs? When was Awk first adopted into Unix?</b></p> <p><b>Brian:</b> Awk was created in 1977, so it was part of 7th-edition Unix, which I think appeared in about 1979. I wouldn't say it was adopted, so much as it was just another program included because it was there. People picked it up very quickly, and we soon had users all over the Labs. People wrote much bigger programs than we had ever anticipated, too, even tens of thousands of lines, which was amazing. But for some kinds of applications, the language was a good match.</p> <p><b>Jim: Has Awk changed over the years, or is Awk today more or less the same Awk from 1977?</b></p> <p><b>Brian:</b> Overall, it's been pretty stable, but there have been a fair number of small things, mostly to keep up with at least the core parts of Gawk. Examples include things like functions to do case conversion, shorthands for some kinds of regular expressions, or special filenames like <code>/dev/stderr</code>. Internally, there's been a lot of work to replace fixed-size arrays with arrays that grow. Arnold Robbins, who maintains Gawk, has also been incredibly helpful with Awk, providing good advice, testing, code, and help with Git.</p> <p><b>Jim: You're currently adding Unicode support to Awk. This is one of those projects that seems obvious when you hear it, because Unicode is everywhere, but not every program supports it yet. Tell us about your project to add Unicode to Awk.</b></p> <p><b>Brian:</b> It's been sort of embarrassing for a while now that Awk only handled 8-bit input, though in fairness it predates Unicode by 10 or 20 years. Gawk, the GNU version, has handled Unicode properly for quite a while, so it's good to be up to date and compatible.</p> <p></p><div class="embedded-resource-list callout-float-right"> <div class="field field--name-title field--type-string field--label-hidden field__item">More Linux resources</div> <div class="field field--name-links field--type-link field--label-hidden field__items"> <div class="field__item"><a href="https://developers.redhat.com/cheat-sheets/linux-commands-cheat-sheet/?intcmp=70160000000h1jYAAQ">Linux commands cheat sheet</a></div> <div class="field__item"><a href="https://developers.redhat.com/cheat-sheets/advanced-linux-commands/?intcmp=70160000000h1jYAAQ">Advanced Linux commands cheat sheet</a></div> <div class="field__item"><a href="https://www.redhat.com/en/services/training/rh024-red-hat-linux-technical-overview?intcmp=70160000000h1jYAAQ">Free online course: RHEL technical overview</a></div> <div class="field__item"><a href="https://opensource.com/downloads/cheat-sheet-networking?intcmp=70160000000h1jYAAQ">Linux networking cheat sheet</a></div> <div class="field__item"><a href="https://opensource.com/downloads/cheat-sheet-selinux?intcmp=70160000000h1jYAAQ">SELinux cheat sheet</a></div> <div class="field__item"><a href="https://opensource.com/downloads/linux-common-commands-cheat-sheet?intcmp=70160000000h1jYAAQ">Linux common commands cheat sheet</a></div> <div class="field__item"><a href="https://opensource.com/resources/what-are-linux-containers?intcmp=70160000000h1jYAAQ">What are Linux containers?</a></div> <div class="field__item"><a href="https://opensource.com/tags/linux?intcmp=70160000000h1jYAAQ">Our latest Linux articles</a></div> </div> </div> <p><b>Jim: How big of a project is adding Unicode support? Did this require many changes to the source code?</b></p> <p><b>Brian:</b> I haven't counted, but it's probably 200 or 300 lines, primarily concentrated in either the regular expression recognizer or in the various built-in functions that have to operate in characters, not bytes, for Unicode input.</p> <p><b>Jim: How far along are you in adding Unicode to Awk?</b></p> <p><b>Brian:</b> There's a branch of the code at GitHub that's pretty up to date. It's been tested, but there's always room for more testing.</p> <p>One thing to mention: It handles UTF-8 input and output, but for Unicode code points, which are not the same thing as Unicode graphemes. This distinction is important but technically very complicated, at least as I understand it. As a simple example, a letter with an accent could be represented as two code points (letter and accent) or as a single character (grapheme). Doing this right, whatever that means, is very hard.</p> <p><b>Jim: In a <a href="https://www.youtube.com/watch?v=GNyQxXw_oMQ" target="_blank">Computerphile video,</a> you mention adding support for comma-separated values (CSV) parsing to Awk. How is that project going?</b></p> <p><b>Brian:</b> While I had my hands in the code again, I did add support for CSV input, since that's another bit of the language that was always clunky. I haven't done anything for CSV output, since that's easy to do with a couple of short functions, but maybe that should be revisited.</p> <p><b>Jim: What kinds of things do you use Awk for in your day-to-day work?</b></p> <p><b>Brian:</b> Everything. Pretty much anything that fiddles text is a target for Awk. Certainly, the Awk program I use most is a simple one to make all lines in a text document the same length. I probably used it 100 times while writing answers to your questions.</p> <p><b>Jim: What's the coolest (or most unusual) thing you have used Awk to do?</b></p> <p><b>Brian:</b> A long time ago, I wrote a C++ program that converted Awk programs into C++ that looked as close to Awk as I could manage, by doing things like overloading brackets for associative arrays. It was never used, but it was a fun exercise.</p> <h2>Further reading</h2> <ul> <li><a href="https://opensource.com/downloads/cheat-sheet-awk-features">Awk cheat sheet</a></li> <li><a href="https://opensource.com/downloads/awk-ebook">A practical guide to learning Awk</a> (eBook)</li> </ul> </div> <div class="clearfix text-formatted field field--name-field-article-subhead field--type-text-long field--label-hidden field__item"><p>Brian Kernighan discusses the scripting tool Awk, from its creation to current work on Unicode support.</p> </div> <div class="field field--name-field-lead-image field--type-entity-reference field--label-hidden field__item"><article class="media media--type-image media--view-mode-caption"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="/sites/default/files/lead-images/coffee_tea_laptop_computer_work_desk.png" width="520" height="292" alt="Person drinking a hat drink at the computer" title="Person drinking a hat drink at the computer" /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p><a href="https://unsplash.com/@jonasleupe?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText" target="_blank" rel="ugc">Jonas Leupe</a> on <a href="https://unsplash.com/s/photos/tea-cup-computer?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText" target="_blank" rel="ugc">Unsplash</a></p> </div> </article> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-hidden field__items"> <div class="field__item"><a href="/tags/programming" hreflang="en">Programming</a></div> </div> <div class="field field--name-field-listicle-title field--type-string field--label-hidden field__item">What to read next</div> <div class="field field--name-field-listicles field--type-entity-reference field--label-hidden field__items"> <div class="field__item"><a href="/article/22/1/interview-brian-kernighan" hreflang="und">Talking digital with Brian Kernighan</a></div> <div class="field__item"><a href="/article/21/1/learn-awk" hreflang="und">Learn awk by coding a &quot;guess the number&quot; game</a></div> </div> <div class="field field--name-field-default-license field--type-list-string field--label-hidden field__item"><a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"> <img alt="Creative Commons License" src="/themes/osdc/assets/img/cc-by-sa-4.png" title="This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License." /></a>This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.</div> <section id='comments' class="field field--name-field-comments field--type-comment field--label-hidden comment-wrapper"> <div class="comments__count"> <div class="login"><a href="/user/register?absolute=1">Register</a> or <a href="/user/login?current=/rss.xml&amp;absolute=1">Login</a> to post a comment.</div> </div> </section> Sat, 01 Oct 2022 07:00:00 +0000 Jim Hall 70210 at https://opensource.com https://opensource.com/article/22/10/whats-new-awk#comments How I dock my Linux laptop https://opensource.com/article/22/9/docking-linux-laptop <span class="field field--name-title field--type-string field--label-hidden">How I dock my Linux laptop</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><a title="View user profile." href="/users/don-watkins" class="username">Don Watkins</a></span> <span class="field field--name-created field--type-created field--label-hidden">Fri, 09/30/2022 - 03:00</span> <div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>Not that long ago, docking a laptop was a new idea to me. Since then, I've set up two different laptops for docking, and I've been more than satisfied with how this configuration works. This article describes how I set up the laptop docking stations and gives tips and guidance for anyone considering the same.</p> <h2>How I discovered docking</h2> <p>I began 2020 with one laptop: a <a href="https://opensource.com/article/19/3/darter-pro-laptop-review">Darter Pro</a> purchased from System76 the year before. It had been more than adequate, and I enjoyed the freedom of movement that toting a laptop gave me. Then, of course, everything changed.</p> <p>Suddenly, I was at home all the time. Instead of an occasional video conference, remote meetings and get-togethers became the norm. I grew tired of the little square images of colleagues on the Darter Pro's 15.6-inch display, and the included 256 GB Non-Volatile Memory express (NVMe) drive was a limited amount of storage for my needs.</p> <p>I purchased an <a href="https://opensource.com/article/20/9/linux-intel-nuc">Intel NUC</a> (Next Unit of Computing) kit and a 27-inch LCD display, rearranged my office with a table, and settled into this new paradigm. Later in the year, I built another, more powerful NUC with an i7 processor, 32 GB of RAM, and a terabyte NVMe drive to accommodate video games and the video converted from old family 8mm movies and VHS tapes.</p> <p>Now I had a laptop and a desktop (both running Linux distributions, of course). I regularly met with colleagues and friends on a variety of video conferencing platforms, and I thoroughly enjoyed the increased desktop real estate afforded by the 27-inch display.</p> <p>Fast-forward 18 months to summer 2022, when I discovered laptop docking stations.</p> <p></p><div class="embedded-resource-list callout-float-right"> <div class="field field--name-title field--type-string field--label-hidden field__item">More Linux resources</div> <div class="field field--name-links field--type-link field--label-hidden field__items"> <div class="field__item"><a href="https://developers.redhat.com/cheat-sheets/linux-commands-cheat-sheet/?intcmp=70160000000h1jYAAQ">Linux commands cheat sheet</a></div> <div class="field__item"><a href="https://developers.redhat.com/cheat-sheets/advanced-linux-commands/?intcmp=70160000000h1jYAAQ">Advanced Linux commands cheat sheet</a></div> <div class="field__item"><a href="https://www.redhat.com/en/services/training/rh024-red-hat-linux-technical-overview?intcmp=70160000000h1jYAAQ">Free online course: RHEL technical overview</a></div> <div class="field__item"><a href="https://opensource.com/downloads/cheat-sheet-networking?intcmp=70160000000h1jYAAQ">Linux networking cheat sheet</a></div> <div class="field__item"><a href="https://opensource.com/downloads/cheat-sheet-selinux?intcmp=70160000000h1jYAAQ">SELinux cheat sheet</a></div> <div class="field__item"><a href="https://opensource.com/downloads/linux-common-commands-cheat-sheet?intcmp=70160000000h1jYAAQ">Linux common commands cheat sheet</a></div> <div class="field__item"><a href="https://opensource.com/resources/what-are-linux-containers?intcmp=70160000000h1jYAAQ">What are Linux containers?</a></div> <div class="field__item"><a href="https://opensource.com/tags/linux?intcmp=70160000000h1jYAAQ">Our latest Linux articles</a></div> </div> </div> <h2>Docking a Linux laptop</h2> <p>I didn't realize I could have docked my Darter Pro using its USB-C port. My appetite for new hardware had me pining for the new <a href="https://opensource.com/article/22/7/hp-dev-one-review">HP Dev One</a>, which comes with Pop!_OS preinstalled, a Ryzen 7 processor, 16 GB of RAM, a terabyte NVMe drive, and two USB-C ports.</p> <p>Opensource.com alumnus <a href="https://opensource.com/users/jlacroix" target="_blank">Jay LeCroix</a> has an <a href="https://www.youtube.com/watch?v=bTfGy0lPPFA" target="_blank">excellent video</a> explaining how to use USB-C docking with Linux laptops. That's when I began to appreciate the utility of USB-C docking hardware. After watching the video and doing some additional research, I decided to purchase a USB-C docking station for the Darter Pro.</p> <p>System76 has <a href="https://support.system76.com/articles/docking-station/" target="_blank">a list of community-recommended docks</a>, including the Plugable UD-CA1A dock. The Plugable dock claims to work with all Intel and NVIDIA systems, so I decided to order a unit. The dock was easy to set up, and once connected to the Darter Pro, I had the best of both worlds. I had the freedom to move when I wanted to and the ability to dock if I wanted to. But I needed more storage space than the 256GB drive in the Darter Pro.</p> <p>My success with the docking station led me to purchase the HP Dev ONE, eager for the increased storage and speed. When it arrived, I quickly set it up. I backed up all the files on the NUC and restored them on my new Dev ONE. The new laptop was easily connected to the docking station.</p> <h2>Docking the Dev ONE</h2> <p>The <a href="https://opensource.com/article/22/7/hp-dev-one-review" target="_blank">Dev ONE</a> has two USB-C ports, and either one connects to the dock. It still amazes me that power and video pass through this single port. In addition to the two USB-A ports on my laptop, I now have the additional ports on the docking station. The Plugable dock comes with a headphone jack, microphone jack, 4K high-definition multimedia interface (HDMI) output, one USB-C port on the rear, three USB-A 3.0 ports in the front, two USB-A 2.0 ports on the back, and a Gigabit Ethernet port. My new laptop has a webcam, but I plug an external webcam into one of the rear USB ports to accommodate video conferencing when I'm docked.</p> <p>I enjoy the flexibility and power that docking has added to my workflow. I use video conferencing daily, and when I do, I'm docked and connected to a 27-inch display. I love how easy it is to transition from one setup to another. If you're using a laptop with a USB-C port, I recommend looking into a good docking station. It's well worth it.</p> <p>Or are you already using a dock with your Linux laptop? What are your USB-C docking station recommendations? Be sure to share your experience in the comments below.</p> </div> <div class="clearfix text-formatted field field--name-field-article-subhead field--type-text-long field--label-hidden field__item"><p>Docking a Linux laptop offers flexibility and a better video experience.</p> </div> <div class="field field--name-field-lead-image field--type-entity-reference field--label-hidden field__item"><article class="media media--type-image media--view-mode-caption"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="/sites/default/files/lead-images/oatmeal-and-fedora.jpg" width="520" height="292" alt="Oatmeal and a laptop." title="Oatmeal and a laptop." /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p>Opensource.com</p> </div> </article> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-hidden field__items"> <div class="field__item"><a href="/tags/linux" hreflang="en">Linux</a></div> <div class="field__item"><a href="/tags/hardware" hreflang="en">Hardware</a></div> </div> <div class="hidden field field--name-field-listicle-title field--type-string field--label-hidden field__item">What to read next</div> <div class="field field--name-field-default-license field--type-list-string field--label-hidden field__item"><a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"> <img alt="Creative Commons License" src="/themes/osdc/assets/img/cc-by-sa-4.png" title="This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License." /></a>This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.</div> <section id='comments' class="field field--name-field-comments field--type-comment field--label-hidden comment-wrapper"> <div class="comments__count"> <div class="login"><a href="/user/register?absolute=1">Register</a> or <a href="/user/login?current=/rss.xml&amp;absolute=1">Login</a> to post a comment.</div> </div> </section> Fri, 30 Sep 2022 07:00:00 +0000 Don Watkins 70219 at https://opensource.com https://opensource.com/article/22/9/docking-linux-laptop#comments How Podman packaging works on Linux https://opensource.com/article/22/9/podman-packages-linux <span class="field field--name-title field--type-string field--label-hidden">How Podman packaging works on Linux</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><a title="View user profile." href="/users/lsm5" class="username">Lokesh Mandvekar</a></span> <span class="field field--name-created field--type-created field--label-hidden">Thu, 09/29/2022 - 03:00</span> <div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>Over the past few months, the <a href="https://developers.redhat.com/cheat-sheets/podman-cheat-sheet?intcmp=7013a000002qLH8AAM" target="_blank">Podman</a> project has been reworking its process for generating Debian and Ubuntu packages. This article outlines the past and present of the Debian packaging work done by the Podman project team. Please note that this article does not refer to the official <a href="https://packages.debian.org/unstable/podman" target="_blank">Debian</a> and <a href="https://packages.ubuntu.com/search?keywords=podman" target="_blank">Ubuntu</a> packages that Reinhard Tartler and team created and maintain.</p> <h2>Debian build process</h2> <p>Long story short, the typical Debian build process involves "Debianizing" an upstream repository. First, a <code>debian</code> subdirectory containing packaging metadata and any necessary patches is added to the upstream repo. Then the <code>dpkg-buildpackage</code> command is run to generate the <code>.deb</code> packages.</p> <h3>Older Debian build process for Podman</h3> <p>Previously, the Debian packages for Podman were generated using this "Debianization" process. A <code>debian</code> directory containing the packaging metadata was added to the Podman source in a <a href="https://gitlab.com/rhcontainerbot/podman/-/tree/debian/debian" target="_blank">separate fork</a>. That fork got rebased for each new upstream Podman release.</p> <p></p><div class="embedded-resource-list callout-float-right"> <div class="field field--name-title field--type-string field--label-hidden field__item">More Linux resources</div> <div class="field field--name-links field--type-link field--label-hidden field__items"> <div class="field__item"><a href="https://developers.redhat.com/cheat-sheets/linux-commands-cheat-sheet/?intcmp=70160000000h1jYAAQ">Linux commands cheat sheet</a></div> <div class="field__item"><a href="https://developers.redhat.com/cheat-sheets/advanced-linux-commands/?intcmp=70160000000h1jYAAQ">Advanced Linux commands cheat sheet</a></div> <div class="field__item"><a href="https://www.redhat.com/en/services/training/rh024-red-hat-linux-technical-overview?intcmp=70160000000h1jYAAQ">Free online course: RHEL technical overview</a></div> <div class="field__item"><a href="https://opensource.com/downloads/cheat-sheet-networking?intcmp=70160000000h1jYAAQ">Linux networking cheat sheet</a></div> <div class="field__item"><a href="https://opensource.com/downloads/cheat-sheet-selinux?intcmp=70160000000h1jYAAQ">SELinux cheat sheet</a></div> <div class="field__item"><a href="https://opensource.com/downloads/linux-common-commands-cheat-sheet?intcmp=70160000000h1jYAAQ">Linux common commands cheat sheet</a></div> <div class="field__item"><a href="https://opensource.com/resources/what-are-linux-containers?intcmp=70160000000h1jYAAQ">What are Linux containers?</a></div> <div class="field__item"><a href="https://opensource.com/tags/linux?intcmp=70160000000h1jYAAQ">Our latest Linux articles</a></div> </div> </div> <h3>Issues with the Debian build process (for an RPM packager)</h3> <p>While a simple rebase would often work, that was not always the case. Usually, the Podman source itself would require patching to make things work for multiple Debian and Ubuntu versions, leading to rebase failures. And failures in a rebase meant failures in automated tasks. Which, in turn, caused a lot of frustration.</p> <p>This same frustration led our team to <a href="https://podman.io/blogs/2022/04/05/ubuntu-2204-lts-kubic.html" target="_blank">retire the Debian packages</a> in the past. When Podman v3.4 officially made its way into Debian 11 and Ubuntu 22.04 LTS (thanks to the amazing Reinhard Tartler), we thought the Podman project could say goodbye to Debian package maintenance.</p> <p>But that wasn't meant to be. Both Debian and Ubuntu are rather conservative in their package update policies, especially in their release and LTS versions. As a result, many Podman users on Debian-based distributions would be stuck with v3.4 for quite a while, perhaps the entire lifetime of the distro version. While users can often install the latest packages from Debian's experimental repository, that's not necessarily convenient for everyone. As a result, many Debian-based users <a href="https://github.com/containers/podman/issues/14302" target="_blank">asked the Podman project</a> for newer packages.</p> <p>If we were to resurrect the Podman project's own Debian packages, we needed the packaging format to be easy to maintain and debug for RPM packagers and also easy to automate, which meant no frequent failures with rebases and patches.</p> <h2>OBS + Debbuild</h2> <p>The <code>debbuild</code> tool, created by <a href="https://fedoraproject.org/wiki/User:Ngompa" target="_blank">Neal Gompa</a> and others, is a set of RPM packaging macros allowing packagers to build Debian packages using Fedora's packaging sources. Conveniently, <code>debbuild</code> packages can easily be added as dependencies to a project hosted on openSUSE's <a href="https://openbuildservice.org/" target="_blank">Open Build Service</a> infrastructure.</p> <p>Here's a snippet of how <code>debbuild</code> support is enabled for Ubuntu 22.04 on the <a href="https://build.opensuse.org/project/show/devel:kubic:libcontainers:unstable" target="_blank">OBS Stable Kubic repository</a>, maintained by the Podman project:</p> <pre> <div class="geshifilter"><div class="text geshifilter-text"> &lt;repository name="xUbuntu_22.04"&gt;<br />     &lt;path project="Ubuntu:debbuild" repository="Ubuntu_22.04"/&gt;<br />     &lt;path project="Ubuntu:22.04" repository="universe"/&gt;<br />     &lt;arch&gt;x86_64&lt;/arch&gt;<br />     &lt;arch&gt;s390x&lt;/arch&gt;<br />     &lt;arch&gt;armv7l&lt;/arch&gt;<br />     &lt;arch&gt;aarch64&lt;/arch&gt;<br />   &lt;/repository&gt;</div></div></pre><p>The complete configuration file is available <a href="https://build.opensuse.org/projects/devel:kubic:libcontainers:unstable/meta" target="_blank">here</a>.</p> <p>In addition to enabling <code>debbuild</code> packages as dependencies, the Fedora packaging sources must be updated with rules to modify the build process for Debian and Ubuntu environments.</p> <p>Here's a snippet of how it's done for Podman:</p> <pre> <div class="geshifilter"><div class="text geshifilter-text">%if "%{_vendor}" == "debbuild"<br /> Packager: Podman Debbuild Maintainers &lt;https://github.com/orgs/containers/teams/podman-debbuild-maintainers&gt;<br /> License: ASL-2.0+ and BSD and ISC and MIT and MPLv2.0<br /> Release: 0%{?dist}<br /> %else<br /> License: ASL 2.0 and BSD and ISC and MIT and MPLv2.0<br /> Release: %autorelease<br /> ExclusiveArch: %{golang_arches}<br /> %endif</div></div></pre><p>The <strong>" %{_vendor}" == "debbuild"</strong> conditional is used in many other places throughout the spec file. For example, in this code sample, it specifies different sets of dependencies and build steps for Fedora and Debian. Also, <code>debbuild</code> allows conditionalizing Debian and Ubuntu versions using the macros <code>{debian}</code> and <code>{ubuntu}</code>, which are familiar to RPM packagers.</p> <p>You can find the updated RPM spec file with all the <code>debbuild</code> changes <a data-saferedirecturl="https://www.google.com/url?q=https://gitlab.com/rhcontainerbot/rpms-openqa/podman/-/blob/rawhide/podman.spec&amp;source=gmail&amp;ust=1664376742237000&amp;usg=AOvVaw0Vh6vIEQ3y2MT5Rv75cTgz" href="https://gitlab.com/rhcontainerbot/rpms-openqa/podman/-/blob/rawhide/podman.spec" target="_blank">here</a>.</p> <p>These two pieces together produce successful Debian package builds on the <a href="https://build.opensuse.org/package/show/devel:kubic:libcontainers:unstable/podman" target="_blank">OBS Unstable Kubic repository</a><u>.</u></p> <p>Using <code>debbuild</code> also ensures that packaging metadata lives in its own separate repository, implying no patching or rebasing hassles with upstream Podman sources.</p> <h2>Usability</h2> <p>At this time, packages are available for Ubuntu 22.04, Debian Testing, and Debian Unstable. We're in talks with the OBS infrastructure maintainers to adjust the Debian 11 and Ubuntu 20.04 build environments, after which we'll also have successful builds for those two environments.</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">$ apt list podman<br /> Listing... Done<br /> podman<span class="sy0">/</span>unknown,now <span class="nu0">4</span>:4.2.0-0ubuntu22.04+obs55.1 amd64 <span class="br0">[</span>installed<span class="br0">]</span><br /> podman<span class="sy0">/</span>unknown <span class="nu0">4</span>:4.2.0-0ubuntu22.04+obs55.1 arm64<br /> podman<span class="sy0">/</span>unknown <span class="nu0">4</span>:4.2.0-0ubuntu22.04+obs54.1 armhf<br /> podman<span class="sy0">/</span>unknown <span class="nu0">4</span>:4.2.0-0ubuntu22.04+obs54.1 s390x</div></div></pre><p>Now, let's talk usability. These packages have been manually verified, and the Podman team has found them to satisfy typical use cases. Users can install these packages as they would any other DEB package. The repository first needs to be enabled, and there are instructions on the <a href="https://podman.io/getting-started/installation" target="_blank">Podman website</a>. However, these packages are not Debian-approved. They haven't gone through the same quality assurance process as official Debian packages. These packages are currently not recommended for production use, and we urge you to exercise caution before proceeding with installation.</p> <h2>Wrap up</h2> <p>The <code>debbuild</code> project allows the Podman project team to generate Debian packages with a few additions to the Fedora packaging sources, making Debian packaging easier to maintain, debug, and automate. It also allows Debian and Ubuntu users to get the latest Podman at the same speed as Fedora users. Podman users on Debian and Ubuntu looking for the latest updates can use our Kubic <em>unstable</em> repository (ideally <em>not on production environments just yet.</em>)</p> <p>We also highly recommend the <code>debbuild</code> and OBS setup to RPM packagers who must provide Debian and Ubuntu packages to their users. It's a diverse selection of tooling, but open source is all about working together.</p> </div> <div class="clearfix text-formatted field field--name-field-article-subhead field--type-text-long field--label-hidden field__item"><p>Get a deep dive into Podman packages for Debian and Ubuntu using Fedora Sources, OBS, and Debbuild.</p> </div> <div class="field field--name-field-lead-image field--type-entity-reference field--label-hidden field__item"><article class="media media--type-image media--view-mode-caption"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="/sites/default/files/lead-images/OSDC_gift_giveaway_box_520x292.png" width="520" height="292" alt="Gift box opens with colors coming out" title="Gift box opens with colors coming out" /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p>Opensource.com</p> </div> </article> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-hidden field__items"> <div class="field__item"><a href="/tags/linux" hreflang="en">Linux</a></div> <div class="field__item"><a href="/tags/kubernetes" hreflang="en">Kubernetes</a></div> </div> <div class="hidden field field--name-field-listicle-title field--type-string field--label-hidden field__item">What to read next</div> <div class="field field--name-field-default-license field--type-list-string field--label-hidden field__item"><a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"> <img alt="Creative Commons License" src="/themes/osdc/assets/img/cc-by-sa-4.png" title="This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License." /></a>This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.</div> <section id='comments' class="field field--name-field-comments field--type-comment field--label-hidden comment-wrapper"> <div class="comments__count"> <div class="login"><a href="/user/register?absolute=1">Register</a> or <a href="/user/login?current=/rss.xml&amp;absolute=1">Login</a> to post a comment.</div> </div> </section> Thu, 29 Sep 2022 07:00:00 +0000 Lokesh Mandvekar 70211 at https://opensource.com https://opensource.com/article/22/9/podman-packages-linux#comments Build an open source project using this essential advice https://opensource.com/article/22/9/build-open-source-project <span class="field field--name-title field--type-string field--label-hidden">Build an open source project using this essential advice</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><a title="View user profile." href="/users/bolajiayodeji" class="username">Bolaji Ayodeji</a></span> <span class="field field--name-created field--type-created field--label-hidden">Wed, 09/28/2022 - 03:00</span> <div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p><a href="https://opensource.com/resources/what-open-source" target="_blank">Open source</a> is a flourishing and beneficial ecosystem that publicly solves problems in communities and industries using software developed through a decentralized model and community contributions. Over the years, this ecosystem has grown in number and strength among hobbyists and professionals alike. It's mainstream now—even proprietary companies use open source to build software.</p> <p>With the ecosystem booming, many developers want to get in and build new open source projects. The question is: How do you achieve that successfully?</p> <p>This article will demystify the lifecycle and structure of open source projects. I want to give you an overview of what goes on inside an open source project and show you how to build a successful and sustainable project based on my personal experience.</p> <h2>A quick introduction to open source</h2> <p>The <a href="https://opensource.org" target="_blank">Open Source Initiative</a> (OSI) provides a formal, detailed definition of open source, but Wikipedia provides a nice summary:</p> <blockquote><p>Open source software is computer software that is released under a license in which the copyright holder grants users the rights to use, study, change, and distribute the software and its source code to anyone and for any purpose.</p> </blockquote> <p>Open source software is public code, usually on the internet, developed either collaboratively by multiple people or by one person. It's about collaborating with people from different regions, cultures, and technical backgrounds, often working remotely. This is why creating a project that welcomes everyone and enables different people to work together is essential.</p> <h2>The anatomy of an open source project</h2> <p>Like the human body, an open source project is made up of several structures that form the entire system. I think of them as two branches: the people (microscopic) and the documents (macroscopic).</p> <h3>Branch one: people</h3> <p>Generally, an open source project includes the following sets of people:</p> <ul> <li><b>Creators:</b> Those who created the project</li> <li><b>Maintainers:</b> Those who actively manage the entire project</li> <li><b>Contributors:</b> Those who contribute to the project (someone like you!)</li> <li><b>User:</b> Those who use the project, including developers and nontechnical customers</li> <li><b>Working group:</b> A collection of contributors split into domain-specific groups to focus on a discussion or activity around a specific subject area (such as documentation, onboarding, testing, DevOps, code reviews, performance, research, and so on)</li> <li><b>Sponsor:</b> Those who contribute financial support to the project</li> </ul> <p>You need to consider each group in the list above as you prepare to build a new project. What plan do you have for each of them?</p> <ul> <li>For maintainers, decide on the criteria you want to use to appoint them. Usually, an active contributor makes the best maintainer.</li> <li>For users and contributors, you want to prepare solid documentation, an onboarding process, and everything else they need to succeed when working with your project.</li> <li>For working groups, decide whether you need them and how your project may be logically split in the future.</li> <li>Finally, for sponsors, you must provide enough data and information about your project to enable them to choose to sponsor you.</li> </ul> <p>You don't need to have all of these figured out at the start of your project. However, it's wise to think about them at the early stages so you can build the right foundations to ensure that future additions stand firm and lead to a successful project.</p> <p></p><div class="embedded-resource-list callout-float-right"> <div class="field field--name-title field--type-string field--label-hidden field__item">More great content</div> <div class="field field--name-links field--type-link field--label-hidden field__items"> <div class="field__item"><a href="https://www.redhat.com/en/services/training/rh024-red-hat-linux-technical-overview?intcmp=7016000000127cYAAQ">Free online course: RHEL technical overview</a></div> <div class="field__item"><a href="https://developers.redhat.com/cheat-sheets/advanced-linux-commands/?intcmp=7016000000127cYAAQ">Learn advanced Linux commands</a></div> <div class="field__item"><a href="https://opensource.com/downloads/cheat-sheets?intcmp=7016000000127cYAAQ">Download cheat sheets</a></div> <div class="field__item"><a href="https://opensource.com/tags/linux?intcmp=7016000000127cYAAQ">Find an open source alternative</a></div> <div class="field__item"><a href="https://opensource.com/resources?intcmp=7016000000127cYAAQ">Explore open source resources</a></div> </div> </div> <h3>Branch two: documents</h3> <p>Open source projects usually include the following documents, usually in plain text or markdown format:</p> <ul> <li><b>License:</b> This legal document explains how and to what extent the project can be freely used, modified, and shared. A list of OSI-approved licenses is available on the OSI website. Without an explicit license, your project is not legally open source!<br />  </li> <li><b>Code of conduct:</b> This document outlines the rules, norms, acceptable practices, and responsibilities of anyone who decides to participate in the project in any way (including what happens when someone violates any of the rules). The <a href="https://www.contributor-covenant.org/version/2/1/code_of_conduct">Contributor Covenant</a> is a good example and is open source (licensed under a Creative Commons license).<br />  </li> <li><b>README: </b>This file introduces your project to newcomers. On many Git hosting websites, such as GitLab, GitHub, and Codeberg, the README file is displayed under the initial file listing of a repository. It's common to feature documentation here, with links to other necessary documents.<br />  </li> <li><b>Documentation:</b> This is a file or directory containing all documentation resources for the project, including guides, API references, tutorials, and so on.<br />  </li> <li><b>Contributing:</b> Include a document explaining how to contribute to the project, including installation guides, configuration, and so on.<br />  </li> <li><b>Security:</b> Include a file explaining how to submit vulnerability reports or security issues.</li> </ul> <p>Additionally, a project usually has web pages for issues, support, and collaboration.</p> <p>Broadly, these include:</p> <ul> <li><b>Issues or bug reports:</b> A place where users can report bugs. This page also provides a place developers can go to assign themselves the task of fixing one or more of them.<br />  </li> <li><b>Pull or merge requests:</b> A place with proposed feature enhancements and solutions to bugs. These patches may be created by anyone, reviewed by the maintainers, then merged into the project's code.<br />  </li> <li><b>Discussions: </b>A place where maintainers, contributors, and users discuss an open source project. This may be a dedicated website or a forum within a collaborative coding site.</li> </ul> <p>Most projects also have a communication channel in the form of an online chat for conversations and interactions between community members.</p> <h2>Licensing</h2> <p><a href="https://opensource.com/tags/licensing" target="_blank">Licensing</a> is perhaps the easiest but most important criterion to consider before creating an open source project. A license defines the terms and conditions that allow the source code and other components of your project to be used, modified, and shared.</p> <p>Licenses contain tons of legal jargon that many people don't fully understand. I use <a href="https://choosealicense.com">choosealicense.com</a>, which helps you choose a license based on your intended community, your desire to get patches back from those using your code, or your willingness to allow people to use your code without sharing improvements they make to it.</p> <article class="align-center media media--type-image media--view-mode-default"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="https://opensource.com/sites/default/files/2022-09/choosealicense.png" width="675" height="626" alt="This provides guidance on when to use an MIT license or a GNU GPLv3 license. It also recommends that people contributing to a community use the license preferred by that community. The graphic also notes that there are many more licenses available. The website choosealicense.com has a text-based version of this image that links to more detailed information. " /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p>(Bolaji Ayodeji, CC BY-SA 4.0)</p> </div> </article> <h2>13 phases of creating an open source project</h2> <p>Now for the essential question: How do you start an open source software project?</p> <p>Here is a list of what I consider the phases of an open source project.</p> <ol> <li>Brainstorm your idea, write a synopsis, and document it properly.<br />  </li> <li>Begin developing your idea. This usually involves figuring out the right tools and stacks to use, writing some code, version controlling the code, debugging, drinking some coffee, hanging around StackOverflow, using other open source projects, sleeping, and building something to solve a defined problem—or just for fun!<br />  </li> <li>Test the project locally, write some unit and integration tests as required, set up <a href="https://opensource.com/article/19/9/intro-building-cicd-pipelines-jenkins">CI/CD pipelines</a> as needed, create a staging branch (a test branch where you test the code live before merging into the main branch), and do anything else you need to deploy the project.<br />  </li> <li>Write <a href="https://slides.com/bolajiayodeji/effective-oss-docs" target="_blank">good and effective documentation</a>. This should cover what your project does, why it is useful, how to get started with it (usage, installation, configuration, contributing), and where people can get support.<br />  </li> <li>Ensure to document all code conventions you want to use. Enforce them with tools like linters, code formatters, Git hooks, and the commitizen command line utility.<br />  </li> <li>Choose the right license and create a README.<br />  </li> <li>Publish the project on the internet (you might have a private repository initially, and make it public at this step).<br />  </li> <li>Set up the processes for making releases and documenting changelogs (you can use tools like Changesets).<br />  </li> <li>Market the project to the world! You can make a post on social media, start a newsletter, share it with your friends privately, do a product hunt launch, live stream, or any other traditional marketing strategy you know.<br />  </li> <li>Seek funding support by using any of the available funding platforms, like Open Collective, GitHub Sponsors, Patreon, Buy me a Coffee, LiberaPay, and so on. When you create accounts with these platforms, add a link to it in your project's documentation and website.<br />  </li> <li>Build a community around your project.<br />  </li> <li>Consider introducing working groups to break your project's management into logical parts when required.<br />  </li> <li>Continuously implement new ideas that sustain the resources and people behind your project.</li> </ol> <p>It's important to measure different parts of your project as you progress. This provides you with data you can use for evaluation and future growth strategies.</p> <h2>Now start a project!</h2> <p>I hope this article helps you move forward with that project you've been thinking about.</p> <p>Feel free to use it as a guide and fill any gaps I missed as you build your awesome open source software project.</p> </div> <div class="clearfix text-formatted field field--name-field-article-subhead field--type-text-long field--label-hidden field__item"><p>Use these steps for a solid foundation for your first—or next—project.</p> </div> <div class="field field--name-field-lead-image field--type-entity-reference field--label-hidden field__item"><article class="media media--type-image media--view-mode-caption"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="/sites/default/files/lead-images/BUSINESS_buildtogether.png" width="520" height="292" alt="People working together to build " title="People working together to build " /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p>Opensource.com</p> </div> </article> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-hidden field__items"> <div class="field__item"><a href="/tags/community-management" hreflang="en">Community management</a></div> </div> <div class="field field--name-field-listicle-title field--type-string field--label-hidden field__item">What to read next</div> <div class="field field--name-field-listicles field--type-entity-reference field--label-hidden field__items"> <div class="field__item"><a href="/article/22/4/open-source-language-tool-lanfrica" hreflang="en">New open source tool catalogs African language resources</a></div> <div class="field__item"><a href="/article/22/9/first-pull-request-merged" hreflang="en">I got my first pull request merged!</a></div> <div class="field__item"><a href="/article/19/12/open-source-contributors" hreflang="und">10 tips for onboarding open source contributors</a></div> </div> <div class="field field--name-field-default-license field--type-list-string field--label-hidden field__item"><a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"> <img alt="Creative Commons License" src="/themes/osdc/assets/img/cc-by-sa-4.png" title="This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License." /></a>This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.</div> <section id='comments' class="field field--name-field-comments field--type-comment field--label-hidden comment-wrapper"> <div class="comments__count"> <h2>1 Comment</h2> <div class="login"><a href="/user/register?absolute=1">Register</a> or <a href="/user/login?current=/rss.xml&amp;absolute=1">Login</a> to post a comment.</div> </div> <article data-comment-user-id="106186" id="comment-219883" class="comment js-comment"> <mark class="hidden" data-comment-timestamp="1664363015"></mark> <footer class="comment__meta"> <article class="media media--type-image media--view-mode-byline"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="/sites/default/files/styles/medium/public/pictures/nicolasverite_0.jpg?itok=Ob9N_W7-" width="190" height="220" class="image-style-medium" /> </div> </article> <div> <div class="comment__submitted"> <a href="/users/nyco">Nicolas Vérité</a> | September 28, 2022 <div class="rate-widget thumbsup"><form class="comment-content-219883-updown-rate-widget-0 vote-form" id="rate-widget-base-form" data-drupal-selector="comment-content-219883-updown-rate-widget-0" action="/feed" method="post" accept-charset="UTF-8"> <div class="rate-widget-thumbs-up"> <div class="thumbsup-rating-wrapper rate-disabled" can-edit="false"><div class="form-item js-form-item form-type-radio js-form-type-radio form-item-value js-form-item-value form-disabled"> <label class="form-item js-form-item form-type-radio js-form-type-radio form-item-value js-form-item-value form-disabled rating-label thumbsup-rating-label thumbsup-rating-label-up vote-pending rate-thumbs-up-btn-up"><input twig-suggestion="rating-input" class="rating-input thumbsup-rating-input form-radio" data-drupal-selector="edit-value-1" disabled="disabled" type="radio" id="edit-value-1" name="value" value="1" /> </label> </div> </div><input autocomplete="off" data-drupal-selector="form-2ag7xeceffwvfovzrp0p8lnezgl8rmmaisxdtqgk29k" type="hidden" name="form_build_id" value="form-2aG7XeCEFFwVFoVZrP0P8lNEZGL8RmmaISxDTqGK29k" /> <input data-drupal-selector="edit-comment-content-219883-updown-rate-widget-0" type="hidden" name="form_id" value="comment_content_219883_updown_rate_widget_0" /> <input class="thumbsup-rating-submit button button--primary js-form-submit form-submit" data-drupal-selector="edit-submit" type="submit" id="edit-submit--2" name="op" value="Save" /> <div class="vote-result js-form-wrapper form-wrapper" data-drupal-selector="edit-result" id="edit-result"> <div title="Register or Login to like."><a href="/user/register">Register</a> or <a href="/user/login?current=/rss.xml">Login</a> to like</div> </div> <div class="rate-score"></div> </div> </form> </div> </div> <div class="content"> <div class="clearfix text-formatted field field--name-comment-body field--type-text-long field--label-hidden field__item"><p>Awesome article, that must be distributed within all companies considering opensourcing some code...</p> <p>That said, I would improve it by adding a section about the target (who the software is for), the problem that it resolves, and the iterative+experimental process.</p> </div> <drupal-render-placeholder callback="flag.link_builder:build" arguments="0=comment&amp;1=219883&amp;2=report_as_spam" token="0cU8E2p64DrcnPoq9MiO3lh0hY52nwuQKcTKNbH4fcg"></drupal-render-placeholder><drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=219883&amp;1=full&amp;2=en&amp;3=" token="4nvmBbMmObymxE7kthwDADYnbkihcy7W2gMonUtSwgk"></drupal-render-placeholder> </div> </div> </footer> </article> </section> Wed, 28 Sep 2022 07:00:00 +0000 Bolaji Ayodeji 70216 at https://opensource.com https://opensource.com/article/22/9/build-open-source-project#comments 165+ JavaScript terms you need to know https://opensource.com/article/22/9/javascript-glossary <span class="field field--name-title field--type-string field--label-hidden">165+ JavaScript terms you need to know</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><a title="View user profile." href="/users/sacsam005" class="username">Sachin Samal</a></span> <span class="field field--name-created field--type-created field--label-hidden">Tue, 09/27/2022 - 03:00</span> <div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>JavaScript is a rich language, with sometimes a seemingly overwhelming number of libraries and frameworks. With so many options available, it's sometimes useful to just look at the language itself and keep in mind its core components. This glossary covers the core JavaScript language, syntax, and functions.</p> <h2>JavaScript variables</h2> <p><code>var</code>: The most used variable. Can be reassigned but only accessed within a function, meaning function scope. Variables defined with <code>var</code> move to the top when code is executed.</p> <p><code>const</code>: Cannot be reassigned and not accessible before they appear within the code, meaning block scope.</p> <p><code>let</code>: Similar to <code>const</code> with block scope, however, the <code>let</code> variable can be reassigned but not re-declared.</p> <h3>Data types</h3> <p>Numbers: <code>var age = 33</code></p> <p>Variables: <code>var a</code></p> <p>Text (strings): <code>var a = "Sachin"</code></p> <p>Operations: <code>var b = 4 + 5 + 6</code></p> <p>True or false statements: <code>var a = true</code></p> <p>Constant numbers: <code>const PI = 3.14</code></p> <p>Objects: <code>var fullName = {firstName:"Sachin", lastName: "Samal"}</code></p> <h3>Objects</h3> <p>This is a simple example of objects in JavaScript. This object describe the variable <code>car</code>, and includes <em>keys</em> or <em>properties</em> such as <code>make</code>, <code>model</code>, and <code>year</code> are the object's property names. Each property has a value, such as <code>Nissan</code>, <code>Altima</code>, and <code>2022</code>. A JavaScript object is a collection of properties with values, and it functions as a method.</p> <pre> <div class="geshifilter"><div class="javascript geshifilter-javascript"><span class="kw1">var</span> car <span class="sy0">=</span> <span class="br0">{</span><br /> make<span class="sy0">:</span><span class="st0">"Nissan"</span><span class="sy0">,</span><br /> model<span class="sy0">:</span><span class="st0">"Altima"</span><span class="sy0">,</span><br /> year<span class="sy0">:</span><span class="nu0">2022</span><span class="sy0">,</span><br /> <span class="br0">}</span><span class="sy0">;</span></div></div></pre><h3>Comparison operators</h3> <p><code>==</code>: Is equal to</p> <p><code>===</code>: Is equal value and equal type</p> <p><code>!=</code>: Is not equal</p> <p><code>!==</code>: Is not equal value or not equal type</p> <p><code>&gt;</code>: Is greater than</p> <p><code>&lt;</code>: Is less than</p> <p><code>&gt;=</code>: Is greater than or equal to</p> <p><code>&lt;=</code>: Is less than or equal to</p> <p><code>?</code>: Ternary operator</p> <h3>Logical operators</h3> <p><code>&amp;&amp;</code>: Logical AND</p> <p><code>||</code>: Logical OR</p> <p><code>!</code>: Logical NOT</p> <h3>Output data</h3> <p><code>alert()</code>: Output data in an alert box in the browser window</p> <p><code>confirm()</code>: Open up a yes/no dialog and return true/false depending on user click</p> <p><code>console.log()</code>: Write information to the browser console. Good for debugging.</p> <p><code>document.write()</code>: Write directly to the HTML document</p> <p><code>prompt()</code>: Create a dialog for user input</p> <h2>Array methods</h2> <p>Array: An object that can hold multiple values at once.</p> <p><code>concat()</code>: Join several arrays into one</p> <p><code>indexOf()</code>: Return the primitive value of the specified object</p> <p><code>join()</code>: Combine elements of an array into a single string and return the string</p> <p><code>lastIndexOf()</code>: Give the last position at which a given element appears in an array</p> <p><code>pop()</code>: Remove the last element of an array</p> <p><code>push()</code>: Add a new element at the end</p> <p><code>reverse()</code>: Sort elements in descending order</p> <p><code>shift()</code>: Remove the first element of an array</p> <p><code>slice()</code>: Pull a copy of a portion of an array into a new array</p> <p><code>splice()</code>: Add positions and elements in a specified way</p> <p><code>toString()</code>: Convert elements to strings</p> <p><code>unshift()</code>: Add a new element to the beginning</p> <p><code>valueOf()</code>: Return the first position at which a given element appears in an array</p> <p></p><div class="embedded-resource-list callout-float-right"> <div class="field field--name-title field--type-string field--label-hidden field__item">Programming and development</div> <div class="field field--name-links field--type-link field--label-hidden field__items"> <div class="field__item"><a href="https://developers.redhat.com/?intcmp=7016000000127cYAAQ">Red Hat Developers Blog</a></div> <div class="field__item"><a href="https://opensource.com/downloads/cheat-sheets?intcmp=7016000000127cYAAQ">Programming cheat sheets</a></div> <div class="field__item"><a href="https://www.redhat.com/en/services/training/learning-subscription?intcmp=7016000000127cYAAQ">Try for free: Red Hat Learning Subscription</a></div> <div class="field__item"><a href="https://opensource.com/downloads/bash-programming-guide?intcmp=7016000000127cYAAQ">eBook: An introduction to programming with Bash</a></div> <div class="field__item"><a href="https://developers.redhat.com/cheat-sheets/bash-shell-cheat-sheet?intcmp=7016000000127cYAAQ">Bash shell scripting cheat sheet</a></div> <div class="field__item"><a href="https://developers.redhat.com/e-books/modernizing-enterprise-java?intcmp=7016000000127cYAAQ">eBook: Modernizing Enterprise Java</a></div> </div> </div> <h2>JavaScript loops</h2> <p>Loops: Perform specific tasks repeatedly under applied conditions.</p> <pre> <div class="geshifilter"><div class="javascript geshifilter-javascript"><span class="kw1">for</span> <span class="br0">(</span>before loop<span class="sy0">;</span> condition <span class="kw1">for</span> loop<span class="sy0">;</span> execute after loop<span class="br0">)</span> <span class="br0">{</span><br /> <span class="co1">// what to do during the loop</span><br /> <span class="br0">}</span></div></div></pre><p><code>for</code>: Creates a conditional loop</p> <p><code>while</code>: Sets up conditions under which a loop executes at least once, as long as the specified condition is evaluated as true</p> <p><code>do while</code>: Similar to the <code>while</code> loop, it executes at least once and performs a check at the end to see if the condition is met. If it is, then it executes again</p> <p><code>break</code>: Stop and exit the cycle at certain conditions</p> <p><code>continue</code>: Skip parts of the cycle if certain conditions are met</p> <h2>if-else statements</h2> <p>An <code>if</code> statement executes the code within brackets as long as the condition in parentheses is true. Failing that, an optional <code>else</code> statement is executed instead.</p> <pre> <div class="geshifilter"><div class="javascript geshifilter-javascript"><span class="kw1">if</span> <span class="br0">(</span>condition<span class="br0">)</span> <span class="br0">{</span><br /> <span class="co1">// do this if condition is met</span><br /> <span class="br0">}</span> <span class="kw1">else</span> <span class="br0">{</span><br /> <span class="co1">// do this if condition is not met</span><br /> <span class="br0">}</span></div></div></pre><h2>Strings</h2> <h3>String methods</h3> <p><code>charAt()</code>: Return a character at a specified position inside a string</p> <p><code>charCodeAt()</code>: Give the Unicode of the character at that position</p> <p><code>concat()</code>: Concatenate (join) two or more strings into one</p> <p><code>fromCharCode()</code>: Return a string created from the specified sequence of UTF-16 code units</p> <p><code>indexOf()</code>: Provide the position of the first occurrence of a specified text within a string</p> <p><code>lastIndexOf()</code>: Same as <code>indexOf()</code> but with the last occurrence, searching backwards</p> <p><code>match()</code>: Retrieve the matches of a string against a search pattern</p> <p><code>replace()</code>: Find and replace specified text in a string</p> <p><code>search()</code>: Execute a search for a matching text and return its position</p> <p><code>slice()</code>: Extract a section of a string and return it as a new string</p> <p><code>split()</code>: Split a string object into an array of strings at a specified position</p> <p><code>substr()</code>: Extract a substring depended on a specified number of characters, similar to <code>slice()</code></p> <p><code>substring()</code>: Can't accept negative indices, also similar to <code>slice()</code></p> <p><code>toLowerCase()</code>: Convert strings to lower case</p> <p><code>toUpperCase()</code>: Convert strings to upper case</p> <p><code>valueOf()</code>: Return the primitive value (that has no properties or methods) of a string object</p> <h3>Number methods</h3> <p><code>toExponential()</code>: Return a string with a rounded number written as exponential notation</p> <p><code>toFixed()</code>: Return the string of a number with a specified number of decimals</p> <p><code>toPrecision()</code>: String of a number written with a specified length</p> <p><code>toString()</code>: Return a number as a string</p> <p><code>valueOf()</code>: Return a number as a number</p> <h3>Math methods</h3> <p><code>abs(a)</code>: Return the absolute (positive) value of <code>a</code></p> <p><code>acos(x)</code>: Arccosine of <code>x</code>, in radians</p> <p><code>asin(x)</code>: Arcsine of <code>x</code>, in radians</p> <p><code>atan(x)</code>: Arctangent of <code>x</code> as a numeric value</p> <p><code>atan2(y,x)</code>: Arctangent of the quotient of its arguments</p> <p><code>ceil(a)</code>: Value of a rounded up to its nearest integer</p> <p><code>cos(a)</code>: Cosine of <code>a</code> (<code>x</code> is in radians)</p> <p><code>exp(a)</code>: Value of Ex</p> <p><code>floor(a)</code>: Value of a rounded down to its nearest integer</p> <p><code>log(a)</code>: Natural logarithm (base E) of <code>a</code></p> <p><code>max(a,b,c…,z)</code>: Return the number with the highest value</p> <p><code>min(a,b,c…,z)</code>: Return the number with the lowest value</p> <p><code>pow(a,b)</code>: <code>a</code> to the power of <code>b</code></p> <p><code>random()</code>: Return a random number between 0 and 1</p> <p><code>round(a)</code>: Value of <code>a</code> rounded to its nearest integer</p> <p><code>sin(a)</code>: Sine of <code>a</code> (<code>a</code> is in radians)</p> <p><code>sqrt(a)</code>: Square root of <code>a</code></p> <p><code>tan(a)</code>: Tangent of an angle</p> <h2>Dealing with dates in JavaScript</h2> <h3>Set dates</h3> <p><code>Date()</code>: Create a new date object with the current date and time</p> <p><code>Date(2022, 6, 22, 4, 22, 11, 0)</code>: Create a custom date object. The numbers represent year, month, day, hour, minutes, seconds, milliseconds. You can omit anything except for year and month.</p> <p><code>Date("2022-07-29")</code>: Date declaration as a string</p> <h3>Pull date and time values</h3> <p><code>getDate()</code>: Day of the month as a number (1-31)</p> <p><code>getDay()</code>: Weekday as a number (0-6)</p> <p><code>getFullYear()</code>: Year as a four-digit number (yyyy)</p> <p><code>getHours()</code>: Hour (0-23)</p> <p><code>getMilliseconds()</code>: Millisecond (0-999)</p> <p><code>getMinutes()</code>: Minute (0-59)</p> <p><code>getMonth()</code>: Month as a number (0-11)</p> <p><code>getSeconds()</code>: Second (0-59)</p> <p><code>getTime()</code>: Milliseconds since January 1, 1970</p> <p><code>getUTCDate()</code>: Day (date) of the month in the specified date according to universal time (also available for day, month, full year, hours, minutes, etc.)</p> <p><code>parse</code>: Parse a string representation of a date and return the number of milliseconds since January 1, 1970</p> <h3>Set part of a date</h3> <p><code>setDate()</code>: Set the day as a number (1-31)</p> <p><code>setFullYear()</code>: Set the year (optionally month and day)</p> <p><code>setHours()</code>: Set the hour (0-23)</p> <p><code>setMilliseconds()</code>: Set milliseconds (0-999)</p> <p><code>setMinutes()</code>: Set the minutes (0-59)</p> <p><code>setMonth()</code>: Set the month (0-11)</p> <p><code>setSeconds()</code>: Set the seconds (0-59)</p> <p><code>setTime()</code>: Set the time (milliseconds since January 1, 1970)</p> <p><code>setUTCDate()</code>: Set the day of the month for a specified date according to universal time (also available for day, month, full year, hours, minutes, etc.)</p> <h2>Dom mode</h2> <h3>Node methods</h3> <p><code>appendChild()</code>: Add a new child node to an element as the last child node</p> <p><code>cloneNode()</code>: Clone an HTML element</p> <p><code>compareDocumentPosition()</code>: Compare the document position of two elements</p> <p><code>getFeature()</code>: Return an object which implements the APIs of a specified feature</p> <p><code>hasAttributes()</code>: Return <strong>true</strong> if an element has any attributes, otherwise <strong>false</strong></p> <p><code>hasChildNodes()</code>: Return <strong>true</strong> if an element has any child nodes, otherwise <strong>false</strong></p> <p><code>insertBefore()</code>: Insert a new child node before a specified, existing child node</p> <p><code>isDefaultNamespace()</code>: Return <strong>true</strong> if a specified <code>namespaceURI</code> is the default, otherwise <strong>false</strong></p> <p><code>isEqualNode()</code>: Check if two elements are equal</p> <p><code>isSameNode()</code>: Check if two elements are the same node</p> <p><code>isSupported()</code>: Return <strong>true</strong> if a specified feature is supported on the element</p> <p><code>lookupNamespaceURI()</code>: Return the <code>namespaceURI</code> associated with a given node</p> <p><code>normalize()</code>: Join adjacent text nodes and removes empty text nodes in an element</p> <p><code>removeChild()</code>: Remove a child node from an element</p> <p><code>replaceChild()</code>: Replace a child node in an element</p> <h3>Element methods</h3> <p><code>getAttribute()</code>: Return the specified attribute value of an element node</p> <p><code>getAttributeNS()</code>: Return string value of the attribute with the specified namespace and name</p> <p><code>getAttributeNode()</code>: Get the specified attribute node</p> <p><code>getAttributeNodeNS()</code>: Return the attribute node for the attribute with the given namespace and name</p> <p><code>getElementsByTagName()</code>: Provide a collection of all child elements with the specified tag name</p> <p><code>getElementsByTagNameNS()</code>: Return a live HTMLCollection of elements with a certain tag name belonging to the given namespace</p> <p><code>hasAttribute()</code>: Return <strong>true</strong> if an element has any attributes, otherwise <strong>false</strong></p> <p><code>hasAttributeNS()</code>: Provide a true/false value indicating whether the current element in a given namespace has the specified attribute</p> <p><code>removeAttribute()</code>: Remove a specified attribute from an element</p> <p><code>lookupPrefix()</code>: Return a <code>DOMString</code> containing the prefix for a given <code>namespaceURI</code>, if present</p> <p><code>removeAttributeNS()</code>: Remove the specified attribute from an element within a certain namespace</p> <p><code>removeAttributeNode()</code>: Take away a specified attribute node and return the removed node</p> <p><code>setAttribute()</code>: Set or change the specified attribute to a specified value</p> <p><code>setAttributeNS()</code>: Add a new attribute or changes the value of an attribute with the given namespace and name</p> <p><code>setAttributeNode()</code>: Set or change the specified attribute node</p> <p><code>setAttributeNodeNS()</code>: Add a new namespaced attribute node to an element</p> <h2>JavaScript events</h2> <h3>Mouse</h3> <p><code>onclick</code>: User clicks on an element</p> <p><code>oncontextmenu</code>: User right-clicks on an element to open a context menu</p> <p><code>ondblclick</code>: User double-clicks on an element</p> <p><code>onmousedown</code>: User presses a mouse button over an element</p> <p><code>onmouseenter</code>: Pointer moves onto an element</p> <p><code>onmouseleave</code>: Pointer moves out of an element</p> <p><code>onmousemove</code>: Pointer moves while it is over an element</p> <p><code>onmouseover</code>: Pointer moves onto an element or one of its children</p> <p><code>setInterval()</code>: Call a function or evaluates an expression at</p> <p><code>oninput</code>: User input on an element</p> <p><code>onmouseup</code>: User releases a mouse button while over an element</p> <p><code>onmouseout</code>: User moves the mouse pointer out of an element or one of its children</p> <p><code>onerror</code>: Happens when an error occurs while loading an external file</p> <p><code>onloadeddata</code>: Media data is loaded</p> <p><code>onloadedmetadata</code>: Metadata (like dimensions and duration) is loaded</p> <p><code>onloadstart</code>: Browser starts looking for specified media</p> <p><code>onpause</code>: Media is paused either by the user or automatically</p> <p><code>onplay</code>: Media is started or is no longer paused</p> <p><code>onplaying</code>: Media is playing after having been paused or stopped for buffering</p> <p><code>onprogress</code>: Browser is in the process of downloading the media</p> <p><code>onratechange</code>: Media play speed changes</p> <p><code>onseeked</code>: User finishes moving/skipping to a new position in the media</p> <p><code>onseeking</code>: User starts moving/skipping</p> <p><code>onstalled</code>: Browser tries to load the media, but it is not available</p> <p><code>onsuspend</code> — Browser is intentionally not loading media</p> <p><code>ontimeupdate</code>: Play position has changed (e.g., because of fast forward)</p> <p><code>onvolumechange</code>: Media volume has changed (including mute)</p> <p><code>onwaiting</code>: Media paused but expected to resume (for example, buffering)</p> </div> <div class="clearfix text-formatted field field--name-field-article-subhead field--type-text-long field--label-hidden field__item"><p>Keep this JavaScript glossary bookmarked to reference variables, methods, strings, and more.</p> </div> <div class="field field--name-field-lead-image field--type-entity-reference field--label-hidden field__item"><article class="media media--type-image media--view-mode-caption"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="/sites/default/files/lead-images/code_javascript.jpg" width="520" height="292" alt="Javascript code close-up with neon graphic overlay" title="Javascript code close-up with neon graphic overlay" /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p>Photo by Jen Wike Huger</p> </div> </article> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-hidden field__items"> <div class="field__item"><a href="/tags/javascript" hreflang="en">JavaScript</a></div> </div> <div class="hidden field field--name-field-listicle-title field--type-string field--label-hidden field__item">What to read next</div> <div class="field field--name-field-default-license field--type-list-string field--label-hidden field__item"><a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"> <img alt="Creative Commons License" src="/themes/osdc/assets/img/cc-by-sa-4.png" title="This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License." /></a>This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.</div> <section id='comments' class="field field--name-field-comments field--type-comment field--label-hidden comment-wrapper"> <div class="comments__count"> <div class="login"><a href="/user/register?absolute=1">Register</a> or <a href="/user/login?current=/rss.xml&amp;absolute=1">Login</a> to post a comment.</div> </div> </section> Tue, 27 Sep 2022 07:00:00 +0000 Sachin Samal 70209 at https://opensource.com https://opensource.com/article/22/9/javascript-glossary#comments Get change alerts from any website with this open source tool https://opensource.com/article/22/9/changedetection-io-open-source-website-changes <span class="field field--name-title field--type-string field--label-hidden">Get change alerts from any website with this open source tool</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><a title="View user profile." href="/users/dgtlmoon" class="username">Leigh Morresi</a></span> <span class="field field--name-created field--type-created field--label-hidden">Tue, 09/27/2022 - 03:00</span> <div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>The year was 2020, and news about COVID-19 came flooding in so quickly that everyone felt completely overwhelmed with similar news articles providing updates with varying degrees of accuracy.</p> <p>But all I needed to know was when my official government guidelines changed. In the end, that's all that mattered to me.</p> <p>Whether the concern is a pandemic or just the latest tech news, keeping ahead of changes in website content can be critical.</p> <p>The <a href="https://github.com/dgtlmoon/changedetection.io" target="_blank">changedetection.io</a> project provides a simple yet highly capable, open source solution for website change detection and notification. It's easy to set up, and it can notify over 70 (and counting) different notification systems, such as Matrix, Mattermost, <a href="https://opensource.com/tags/nextcloud" target="_blank">Nextcloud</a>, <a href="https://opensource.com/article/19/10/secure-private-messaging" target="_blank">Signal</a>, <a href="https://opensource.com/article/22/3/open-source-chat-zulip" target="_blank">Zulip</a>, <a href="https://opensource.com/article/20/11/home-assistant" target="_blank">Home Assistant</a>, email, and more. It also notifies proprietary applications like Discord, Office365, Reddit, Telegram, and many others.</p> <p>But <a href="https://github.com/dgtlmoon/changedetection.io" target="_blank">changedetection.io</a> isn't just limited to watching web page content. You can also monitor XML and JSON feeds, and it will build an RSS feed of the websites that changed.</p> <p></p><div class="embedded-resource-list callout-float-right"> <div class="field field--name-title field--type-string field--label-hidden field__item">More great content</div> <div class="field field--name-links field--type-link field--label-hidden field__items"> <div class="field__item"><a href="https://www.redhat.com/en/services/training/rh024-red-hat-linux-technical-overview?intcmp=7016000000127cYAAQ">Free online course: RHEL technical overview</a></div> <div class="field__item"><a href="https://developers.redhat.com/cheat-sheets/advanced-linux-commands/?intcmp=7016000000127cYAAQ">Learn advanced Linux commands</a></div> <div class="field__item"><a href="https://opensource.com/downloads/cheat-sheets?intcmp=7016000000127cYAAQ">Download cheat sheets</a></div> <div class="field__item"><a href="https://opensource.com/tags/linux?intcmp=7016000000127cYAAQ">Find an open source alternative</a></div> <div class="field__item"><a href="https://opensource.com/resources?intcmp=7016000000127cYAAQ">Explore open source resources</a></div> </div> </div> <p>Thanks to its built-in JSON simple storage system, there's no need to set up complicated databases to receive and store information. You can <a href="https://github.com/dgtlmoon/changedetection.io#docker" target="_blank">run it as a Docker image</a> or install it with <code>pip</code>. The project has an <a href="https://github.com/dgtlmoon/changedetection.io/wiki" target="_blank">extensive wiki help section</a>, and most common questions are covered there.</p> <p>For sites using complex JavaScript, you can connect your changedetection.io installation to a Chromium or Chrome browser with the built-in <a href="https://github.com/dgtlmoon/changedetection.io/wiki/Playwright-content-fetcher" target="_blank">Playwright content fetcher</a>.</p> <p>Once running, access the application in your browser (<code>http://localhost:5000</code>, by default). You can set a password in the <strong>Settings</strong> section if your computer can be reached from an outside network.</p> <article class="align-center media media--type-image media--view-mode-default"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="https://opensource.com/sites/default/files/2022-09/screenshot.png" width="1296" height="677" alt="change detection watch list" /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p>(Leigh Morresi, CC BY-SA 4.0)</p> </div> </article> <p>Submit the URL of a page you want to monitor. There are several settings related to how the page is filtered. For example, you more than likely do not want to know when a company's stock price listed in their site footer has changed, but you may want to know when they post a news article to their blog.</p> <h2>Monitor a site</h2> <p>Imagine you want to add your favorite website, Opensource.com, to be monitored. You only want to know when the main call-out article contains the word "python" and you want to be notified over Matrix.</p> <p>To do this, begin with the <strong>visual-selector</strong> tool. (This requires the <strong>playwright</strong> browser interface to be connected.)</p> <article class="align-center media media--type-image media--view-mode-default"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="https://opensource.com/sites/default/files/2022-09/changedetect-osdc.png" width="1066" height="750" alt="Find an element to monitor" /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p>(Leigh Morresi, CC BY-SA 4.0)</p> </div> </article> <p>The <strong>visual-selector</strong> tool automatically calculates the best Xpath or CSS filter to target the content. Otherwise, you would get a lot of noise from the daily page updates.</p> <p>Next, visit the <strong>Filters &amp; Triggers</strong> tab.</p> <article class="align-center media media--type-image media--view-mode-default"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="https://opensource.com/sites/default/files/2022-09/changedetect-filters-triggers.webp" width="1059" height="794" alt="Filters and triggers" /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p>(Leigh Morresi, CC BY-SA 4.0)</p> </div> </article> <p>In <strong>CSS/JSON/XPATH Filter</strong> field (the blue circle), you can see the automatically generated CSS filter from the previous step.</p> <p>There are several useful filters available, such as <strong>Remove elements</strong> (good for removing noisy elements), <strong>Ignore text, Trigger/wait for text,</strong> and <strong>Block change-detection if text matches</strong> (used for waiting for some text to disappear, like "sold out").</p> <p>In <strong>Trigger/wait for text</strong> (the red circle), type in the keyword you want to monitor for. (That's "python" in this example.)</p> <p>The final step is in the <strong>Notifications</strong> tab, where you configure where you want to receive your notification. Below I added a Matrix room as the notification target, using the Matrix API.</p> <article class="align-center media media--type-image media--view-mode-default"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="https://opensource.com/sites/default/files/2022-09/step3-notification-matrix.png" width="1309" height="418" alt="Notifications tab" /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p>(Leigh Morresi, CC BY-SA 4.0)</p> </div> </article> <p>The notification URL is in the format of <code>matrixs://username:password@matrix.org/#/room/#room-name:matrix.org</code></p> <p>However, <a href="https://t2bot.io/" target="_blank">t2Bot</a> format is also supported. Here are more <a href="https://github.com/caronc/apprise/wiki/Notify_matrix" target="_blank">Matrix notification options</a>.</p> <p>And that's it! Now you'll receive a message over Matrix whenever the content changes.</p> <h2>There's more</h2> <p>There's so much more to changedetection.io. If you prefer calling a custom JSON API, you don't have to use an API for notifications (use <code>jsons://</code>). You can also create a custom HTTP request (POST and GET), execute JavaScript before checking (perhaps to pre-fill a username and password login field), and many more interesting features, with more to come.</p> <p>Stop browsing the web and start watching the web instead!</p> </div> <div class="clearfix text-formatted field field--name-field-article-subhead field--type-text-long field--label-hidden field__item"><p>Use changedetection.io to get alerts when a website makes changes or updates.</p> </div> <div class="field field--name-field-lead-image field--type-entity-reference field--label-hidden field__item"><article class="media media--type-image media--view-mode-caption"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="/sites/default/files/lead-images/browser_desktop_website_checklist_metrics.png" width="520" height="292" alt="Browser of things" title="Browser of things" /> </div> </article> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-hidden field__items"> <div class="field__item"><a href="/tags/tools" hreflang="en">Tools</a></div> <div class="field__item"><a href="/tags/web-development" hreflang="en">Web development</a></div> </div> <div class="hidden field field--name-field-listicle-title field--type-string field--label-hidden field__item">What to read next</div> <div class="field field--name-field-default-license field--type-list-string field--label-hidden field__item"><a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"> <img alt="Creative Commons License" src="/themes/osdc/assets/img/cc-by-sa-4.png" title="This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License." /></a>This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.</div> <section id='comments' class="field field--name-field-comments field--type-comment field--label-hidden comment-wrapper"> <div class="comments__count"> <div class="login"><a href="/user/register?absolute=1">Register</a> or <a href="/user/login?current=/rss.xml&amp;absolute=1">Login</a> to post a comment.</div> </div> </section> Tue, 27 Sep 2022 07:00:00 +0000 Leigh Morresi 70203 at https://opensource.com https://opensource.com/article/22/9/changedetection-io-open-source-website-changes#comments OpenSSF: on a mission to improve security of open source software https://opensource.com/article/22/9/openssf-open-source-software-security <span class="field field--name-title field--type-string field--label-hidden">OpenSSF: on a mission to improve security of open source software</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><a title="View user profile." href="/users/gkamathe" class="username">Gaurav Kamathe</a></span> <span class="field field--name-created field--type-created field--label-hidden">Mon, 09/26/2022 - 03:00</span> <div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>Open source software (OSS), once a niche segment of the development landscape, is now ubiquitous. This growth is fantastic for the open source community. However, as the usage of OSS increases, so do concerns about security. Especially in mission-critical applications— think medical devices, automobiles, space flight, and nuclear facilities—securing open source technology is of the utmost priority. No individual entity, whether developers, organizations, or governments, can single-handedly solve this problem. The best outcome is possible when all of them come together to collaborate.</p> <p>The <a href="https://openssf.org/" target="_blank">Open Source Security Foundation</a> (OpenSSF) formed to facilitate this collaboration. OpenSSF is best described in its own words:</p> <blockquote><p>The OpenSSF is a cross-industry collaboration that brings together leaders to improve the security of open source software by building a broader community with targeted initiatives and best practices.</p> </blockquote> <h2>Vision</h2> <p>The technical vision of OpenSSF is to handle security proactively, by default. Developers are rightly at the center of this vision. OpenSSF seeks to empower developers to learn secure development practices and automatically receive guidance on them through the day-to-day tools they use. Researchers who identify security issues can send this information backward through the supply chain to someone who can rapidly address the issue. Auditors and regulators are encouraged to devise security policies that can be easily enforced via tooling, and community members provide information on the components they use and test regularly.</p> <h2>Mobilization plan</h2> <p>OpenSSF drafted a mobilization plan based on input from open source developers and leaders from US federal agencies. The result is a set of high-impact actions aimed at improving the resiliency and security of open source software. Based on this plan, 10 streams of investments have been identified, including security education, risk assessment, memory safety, and supply chain improvement. While discussion of these issues is widespread, OpenSSF is the platform that has collected and prioritized these concerns over others to ensure a secure open source ecosystem.</p> <h2>Working groups</h2> <p>Because the 10 streams of investments are quite diverse, OpenSSF is divided into multiple working groups. This strategy allows individual teams to focus on a specific area of expertise and move forward without getting bogged down with more general concerns. The working groups have something for everyone: Developers can contribute to security tooling, maintainers can handle software repositories, and others can contribute by educating developers on best practices, identifying metrics for open source projects, or identifying and securing the critical projects that form the core of the OSS ecosystem.</p> <p></p><div class="embedded-resource-list callout-float-right"> <div class="field field--name-title field--type-string field--label-hidden field__item">More on security</div> <div class="field field--name-links field--type-link field--label-hidden field__items"> <div class="field__item"><a href="https://developers.redhat.com/articles/defensive-coding-guide/?intcmp=7016000000127cYAAQ">The defensive coding guide</a></div> <div class="field__item"><a href="https://www.redhat.com/en/resources/container-security-openshift-cloud-devops-whitepaper?intcmp=7016000000127cYAAQ">10 layers of Linux container security</a></div> <div class="field__item"><a href="https://developers.redhat.com/books/selinux-coloring-book?intcmp=7016000000127cYAAQ">SELinux coloring book</a></div> <div class="field__item"><a href="https://opensource.com/tags/security?intcmp=7016000000127cYAAQ">More security articles</a></div> </div> </div> <h2>Industry participation</h2> <p>Multiple software vendors have become members of OpenSSF in their own capacity. These vendors are important players in the IT ecosystem, ranging from cloud service providers and operating system vendors to companies hosting OSS repositories, creating security tooling, creating computing hardware, and more. The benefit is getting inputs from a variety of sources that others might not be aware of and then collaboratively working on those issues.</p> <h2>Getting involved</h2> <p>There are a variety of ways to participate in the OpenSSF initiative based on your expertise and the amount of time you can set aside for it:</p> <ul> <li>Sign up for their <a href="https://lists.openssf.org/g/openssf-announcements/" target="_blank">mailing list</a> to follow the latest updates and discussions and update your calendar with OpenSSF meetings.</li> <li>If you are looking for more interactive communication, consider joining their <a href="https://slack.openssf.org/" target="_blank">Slack channel</a>.</li> <li>Browse through their past meetings on their <a href="https://www.youtube.com/c/OpenSSF/videos" target="_blank">YouTube channel</a>.</li> <li>Organizations can consider becoming a <a href="https://openssf.org/join/" target="_blank">member</a> of OpenSSF.</li> <li>Developers can quickly look up the <a href="https://github.com/orgs/ossf/repositories" target="_blank">GitHub repo</a> for the software projects they are working on.</li> <li>Most important, consider joining a working group of your choice and make a difference.</li> </ul> <h2>Conclusion</h2> <p>The security industry is growing and needs active participation from the open source community. If you are starting out or wish to specialize in security, OpenSSF provides a platform to work on the right problems in the security space under the guidance of experienced peers in security.</p> </div> <div class="clearfix text-formatted field field--name-field-article-subhead field--type-text-long field--label-hidden field__item"><p>Developers, businesses, and government agencies are working together to ensure the security of open source software, and you can join them.</p> </div> <div class="field field--name-field-lead-image field--type-entity-reference field--label-hidden field__item"><article class="media media--type-image media--view-mode-caption"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="/sites/default/files/lead-images/security-lock-cloud-safe.png" width="520" height="292" alt="lock on world map" title="lock on world map" /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p><a href="https://pixabay.com/en/safety-encryption-ssl-world-2890768/" target="_blank" rel="ugc">Tumisu</a>. <a href="https://creativecommons.org/share-your-work/public-domain/cc0/" target="_blank" rel="ugc">CC0</a></p> </div> </article> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-hidden field__items"> <div class="field__item"><a href="/tags/security" hreflang="en">Security and privacy</a></div> </div> <div class="hidden field field--name-field-listicle-title field--type-string field--label-hidden field__item">What to read next</div> <div class="field field--name-field-default-license field--type-list-string field--label-hidden field__item"><a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"> <img alt="Creative Commons License" src="/themes/osdc/assets/img/cc-by-sa-4.png" title="This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License." /></a>This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.</div> <section id='comments' class="field field--name-field-comments field--type-comment field--label-hidden comment-wrapper"> <div class="comments__count"> <div class="login"><a href="/user/register?absolute=1">Register</a> or <a href="/user/login?current=/rss.xml&amp;absolute=1">Login</a> to post a comment.</div> </div> </section> Mon, 26 Sep 2022 07:00:00 +0000 Gaurav Kamathe 70212 at https://opensource.com https://opensource.com/article/22/9/openssf-open-source-software-security#comments The story behind Joplin, the open source note-taking app https://opensource.com/article/22/9/joplin-interview <span class="field field--name-title field--type-string field--label-hidden">The story behind Joplin, the open source note-taking app</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><a title="View user profile." href="/users/20i" class="username">Richard Chambers</a></span> <span class="field field--name-created field--type-created field--label-hidden">Mon, 09/26/2022 - 03:00</span> <div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>In this interview, I met up with Laurent Cozic, creator of the note-taking app, Joplin. <a href="https://joplinapp.org/">Joplin</a> was a winner of the <a href="https://www.20i.com/foss-awards/winners" target="_blank">20i</a> rewards, so I wanted to find out what makes it such a success, and how he achieved it.</p> <p id="could-you-summarize-what-joplin-does"><strong>Could you summarize what Joplin does?</strong></p> <p><a href="https://opensource.com/article/19/1/productivity-tool-joplin" target="_blank">Joplin</a> is an open source note-taking app. It allows you to capture your thoughts and securely access them from any device.</p> <p id="obviously-there-are-other-note-taking-apps-out-there—but-apart-from-it-being-free-to-use-what-makes-it-different"><strong>Obviously, there are other note-taking apps out there—but apart from it being free to use, what makes it different?</strong></p> <p>The fact that it is open source is an important aspect for many of our users, because it means there is no vendor locking on the data, and that data can be easily exported and accessed in various ways.</p> <p>We also focus on security and data privacy, in particular with the synchronization end-to-end encryption feature, and by being transparent about any connection that the application makes. We also work with security researchers to keep the app more secure.</p> <p>Finally, Joplin can be customized in several different ways—through plugins, which can add new functionalities, and themes to customize the app appearance. We also expose a data API, which allows third-party applications to access Joplin data.</p> <p><strong>[ Related read <a href="https://opensource.com/article/22/8/note-taking-apps-linux" target="_blank">5 note-taking apps for Linux</a> ]</strong></p> <p id="it's-a-competitive-market-so-what-inspired-you-to-build-it"><strong>It's a competitive market, so what inspired you to build it?</strong></p> <p>It happened organically. I started looking into it in 2016, as I was looking at existing commercial note-taking applications, and I didn't like that the notes, attachments, or tags could not easily be exported or manipulated by other tools.</p> <p>This is probably due to vendor locking and partly a lack of motivation from the vendor since they have no incentive to help users move their data to other apps. There is also an issue with the fact that these companies usually will keep the notes in plain text, and that can potentially cause issues in terms of data privacy and security.</p> <p>So I decided to start creating a simple mobile and terminal application with sync capabilities to have my notes easily accessible on my devices. Later the desktop app was created and the project grew from there.</p> <article class="align-center media media--type-image media--view-mode-default"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="https://opensource.com/sites/default/files/2022-09/joplin-chrome-os.png" width="1366" height="768" alt="Image of Joplin on Chrome OS." /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p>(Opensource.com, CC BY-SA 4.0)</p> </div> </article> <p id="how-long-did-joplin-take-to-make"><strong>How long did Joplin take to make?</strong></p> <p>I've been working on it on and off since 2016 but it wasn't full time. The past two years I've been focusing more on it.</p> <p id="what-advice-might-you-have-for-someone-setting-to-create-their-own-open-source-app"><strong>What advice might you have for someone setting to create their own open source app?</strong></p> <p>Pick a project you use yourself and technologies you enjoy working with.</p> <p>Managing an open source project can be difficult sometimes so there has to be this element of fun to make it worthwhile. Then I guess "release early, release often" applies here, so that you can gauge user's interest and whether it makes sense to spend time developing the project further.</p> <p id="how-many-people-are-involved-in-joplin's-development"><strong>How many people are involved in Joplin's development?</strong></p> <p>There are 3-4 people involved in the development. At the moment we also have six students working on the project as part of Google Summer of Code.</p> <p><strong>[ Also read <a href="https://opensource.com/article/21/10/google-summer-code" target="_blank">Our journey to open source during Google Summer of Code</a> ]</strong></p> <p id="lots-of-people-create-open-source-projects-yet-joplin-has-been-a-resounding-success-for-you.-could-you-offer-creators-any-tips-on-how-to-get-noticed"><strong>Lots of people create open source projects, yet Joplin has been a resounding success for you. Could you offer creators any tips on how to get noticed?</strong></p> <p>There's no simple formula and to be honest I don't think I could replicate the success in a different project! You've got to be passionate about what you're doing but also be rigorous, be organized, make steady progress, ensure the code quality remains high, and have a lot of test units to prevent regressions.</p> <p>Also be open to the user feedback you receive, and try to improve the project based on it.</p> <p>Once you've got all that, the rest is probably down to luck—if it turns out you're working on a project that interests a lot of people, things might work out well!</p> <p id="once-you-get-noticed-how-do-you-keep-that-momentum-going-if-you-don't-have-a-traditional-marketing-budget"><strong>Once you get noticed, how do you keep that momentum going, if you don't have a traditional marketing budget?</strong></p> <p>I think it's about listening to the community around the project. For example I never planned to have a forum but someone suggested it on GitHub, so I made one and it became a great way to share ideas, discuss features, provide support, and so on. The community is generally welcoming of newcomers too, which creates a kind of virtuous circle.</p> <p>Next to this, it's important to communicate regularly about the project.</p> <p>We don't have a public roadmap, because the ETA for most features is generally "I don't know", but I try to communicate about coming features, new releases, and so on. We also communicate about important events, the Google Summer of Code in particular, or when we have the chance to win something like the 20i FOSS Awards.</p> <p>Finally, very soon we'll have an in-person meetup in London, which is another way to keep in touch with the community and collaborators.</p> <p id="how-does-user-feedback-influence-the-roadmap"><strong>How does user feedback influence the roadmap?</strong></p> <p>Significantly. Contributors will often work on something simply because they need the feature. But next to this, we also keep track of the features that seem most important to users, based on what we read about on the forum and on the GitHub issue tracker.</p> <p>For example, the mobile app is now high priority because we frequently hear from users that its limitations and issues are a problem to effectively use Joplin.</p> <article class="align-center media media--type-image media--view-mode-default"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="https://opensource.com/sites/default/files/2022-09/joplin-desktop.png" width="1920" height="1080" alt="Image of Joplin being used on a Desktop." /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p>(Opensource.com, CC BY-SA 4.0)</p> </div> </article> <p id="how-do-you-keep-up-to-date-with-the-latest-in-devcoding"><strong>How do you keep up to date with the latest in dev and coding?</strong></p> <p>Mostly by reading Hacker News!</p> <p id="do-you-have-personal-favorite-foss-that-you'd-recommend"><strong>Do you have a personal favorite FOSS that you'd recommend?</strong></p> <p>Among the less well-known projects, <a href="https://heldercorreia.bitbucket.io/speedcrunch/">SpeedCrunch</a> is very good as a calculator. It has a lot of features and it's great how it keeps a history of all previous calculations.</p> <p>I also use <a href="https://opensource.com/article/18/12/keepassx-security-best-practices" target="_blank">KeepassXC</a> as a password manager. It has been improving steadily over the past few years.</p> <p>Finally, <a href="https://opensource.com/article/20/6/open-source-alternatives-vs-code" target="_blank">Visual Studio Code</a> is great as a cross-platform text editor.</p> <p></p><div class="embedded-resource-list callout-float-right"> <div class="field field--name-title field--type-string field--label-hidden field__item">More Linux resources</div> <div class="field field--name-links field--type-link field--label-hidden field__items"> <div class="field__item"><a href="https://developers.redhat.com/cheat-sheets/linux-commands-cheat-sheet/?intcmp=70160000000h1jYAAQ">Linux commands cheat sheet</a></div> <div class="field__item"><a href="https://developers.redhat.com/cheat-sheets/advanced-linux-commands/?intcmp=70160000000h1jYAAQ">Advanced Linux commands cheat sheet</a></div> <div class="field__item"><a href="https://www.redhat.com/en/services/training/rh024-red-hat-linux-technical-overview?intcmp=70160000000h1jYAAQ">Free online course: RHEL technical overview</a></div> <div class="field__item"><a href="https://opensource.com/downloads/cheat-sheet-networking?intcmp=70160000000h1jYAAQ">Linux networking cheat sheet</a></div> <div class="field__item"><a href="https://opensource.com/downloads/cheat-sheet-selinux?intcmp=70160000000h1jYAAQ">SELinux cheat sheet</a></div> <div class="field__item"><a href="https://opensource.com/downloads/linux-common-commands-cheat-sheet?intcmp=70160000000h1jYAAQ">Linux common commands cheat sheet</a></div> <div class="field__item"><a href="https://opensource.com/resources/what-are-linux-containers?intcmp=70160000000h1jYAAQ">What are Linux containers?</a></div> <div class="field__item"><a href="https://opensource.com/tags/linux?intcmp=70160000000h1jYAAQ">Our latest Linux articles</a></div> </div> </div> <p id="i'd-assumed-that-joplin-was-named-after-janis-but-wikipedia-tells-me-its-scott-joplin.-what-made-you-choose-the-name"><strong>I'd assumed that Joplin was named after Janis, but Wikipedia tells me it's Scott Joplin. What made you choose the name?</strong></p> <p>I wanted to name it "jot-it" at first but I think the name was already taken.</p> <p>Since I was listening to Scott Joplin ragtime music a lot back then (I was pretty much obsessed with it), I decided to use his name.</p> <p>I think the meaning of a product name is not too important, as long as the name itself is easy to write, pronounce, remember, and perhaps is associated with something positive (or at least nothing negative).</p> <p>And I think "Joplin" ticks all these boxes.</p> <p id="is-there-anything-you-can-say-about-plans-for-joplin-an-exclusive-tease-of-a-new-feature-perhaps"><strong>Is there anything you can say about plans for Joplin? An exclusive tease of a new feature, perhaps?</strong></p> <p>As mentioned earlier, we are very keen to make improvements to the mobile app, both in terms of UX design and new features.</p> <p>We're also looking at creating a "Plugin Store" to make it easier to browse and install plugins.</p> <p id="thanks-for-your-time-laurent-–-best-of-luck-with-the-future-of-joplin."><strong>Thanks for your time Laurent— best of luck with the future of Joplin.</strong></p> <hr /> <p><em>This interview was originally published on the <a href="https://www.20i.com/blog/joplin-creator-laurent-cozic/" target="_blank">20i blog</a> and has been republished with permission.</em></p> </div> <div class="clearfix text-formatted field field--name-field-article-subhead field--type-text-long field--label-hidden field__item"><p>Laurent Cozic sat down with me to discuss how Joplin got started and what&#039;s next for the open source note-taking app.</p> </div> <div class="field field--name-field-lead-image field--type-entity-reference field--label-hidden field__item"><article class="media media--type-image media--view-mode-caption"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="/sites/default/files/lead-images/wfh_work_home_laptop_work.png" width="1041" height="584" alt="Working from home at a laptop" title="Working from home at a laptop" /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p>Opensource.com</p> </div> </article> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-hidden field__items"> <div class="field__item"><a href="/tags/art-and-design" hreflang="en">Art and design</a></div> </div> <div class="hidden field field--name-field-listicle-title field--type-string field--label-hidden field__item">What to read next</div> <div class="field field--name-field-default-license field--type-list-string field--label-hidden field__item"><a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"> <img alt="Creative Commons License" src="/themes/osdc/assets/img/cc-by-sa-4.png" title="This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License." /></a>This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.</div> <section id='comments' class="field field--name-field-comments field--type-comment field--label-hidden comment-wrapper"> <div class="comments__count"> <div class="login"><a href="/user/register?absolute=1">Register</a> or <a href="/user/login?current=/rss.xml&amp;absolute=1">Login</a> to post a comment.</div> </div> </section> Mon, 26 Sep 2022 07:00:00 +0000 Richard Chambers 70208 at https://opensource.com https://opensource.com/article/22/9/joplin-interview#comments Drop your database for PostgreSQL https://opensource.com/article/22/9/drop-your-database-for-postgresql <span class="field field--name-title field--type-string field--label-hidden">Drop your database for PostgreSQL</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><a title="View user profile." href="/users/seth" class="username">Seth Kenlon</a></span> <span class="field field--name-created field--type-created field--label-hidden">Sat, 09/24/2022 - 03:00</span> <div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>Databases are tools to store information in an organized but flexible way. A spreadsheet is essentially a database, but the constraints of a graphical application render most spreadsheet applications useless to programmers. With <a href="https://www.redhat.com/en/topics/edge-computing/what-is-edge-computing?intcmp=7013a000002qLH8AAM" target="_blank">Edge</a> and IoT devices becoming significant target platforms, developers need powerful but lightweight solutions for storing, processing, and querying large amounts of data. One of my favourite combinations is the PostgreSQL database and <a href="https://github.com/arcapos/luapgsql">Lua bindings</a>, but the possibilities are endless. Whatever language you use, Postgres is a great choice for a database, but you need to know some basics before adopting it.</p> <h2 id="_install_postgres">Install Postgres</h2> <p>To install PostgreSQL on Linux, use your software repository. On Fedora, CentOS, Mageia, and similar:</p> <pre class="highlight"> <span class="geshifilter"><code class="bash geshifilter-bash"><span class="co4">$ </span><span class="kw2">sudo</span> dnf <span class="kw2">install</span> postgresql postgresql-server</code></span></pre><p>On Debian, Linux Mint, Elementary, and similar:</p> <pre class="highlight"> <span class="geshifilter"><code class="bash geshifilter-bash"><span class="co4">$ </span><span class="kw2">sudo</span> apt <span class="kw2">install</span> postgresql postgresql-contrib</code></span></pre><p>On macOS and Windows, download an installer from <a href="https://www.postgresql.org/download/">postgresql.org</a>.</p> <h2 id="_setting_up_postgres">Setting up Postgres</h2> <p>Most distributions install the Postgres database without <em>starting</em> it, but provide you with a script or <a href="https://opensource.com/article/21/4/sysadmins-love-systemd">systemd service</a> to help it start reliably. However, before you start PostgreSQL, you must create a database cluster.</p> <h3 id="_fedora">Fedora</h3> <p>On Fedora, CentOS, or similar, there's a Postgres setup script provided in the Postgres package. Run this script for easy configuration:</p> <pre class="highlight"> <div class="geshifilter"><div class="bash geshifilter-bash">$ <span class="kw2">sudo</span> <span class="sy0">/</span>usr<span class="sy0">/</span>bin<span class="sy0">/</span>postgresql-setup <span class="re5">--initdb</span><br /> <span class="br0">[</span><span class="kw2">sudo</span><span class="br0">]</span> password:<br />  <span class="sy0">*</span> Initializing database <span class="kw1">in</span> <span class="st_h">'/var/lib/pgsql/data'</span><br />  <span class="sy0">*</span> Initialized, logs are <span class="kw1">in</span> <span class="sy0">/</span>var<span class="sy0">/</span>lib<span class="sy0">/</span>pgsql<span class="sy0">/</span>initdb_postgresql.log</div></div></pre><h3 id="_debian">Debian</h3> <p>On Debian-based distributions, setup is performed automatically by <code>apt</code> during installation.</p> <h3 id="_everything_else">Everything else</h3> <p>Finally, if you're running something else, then you can just use the toolchain provided by Postgres itself. The <code>initdb</code> command creates a database cluster, but you must run it as the <code>postgres</code> user, an identity you may temporarily assume using <code>sudo</code>:</p> <pre class="highlight"> <div class="geshifilter"><div class="bash geshifilter-bash">$ <span class="kw2">sudo</span> <span class="re5">-u</span> postgres \<br /> <span class="st0">"initdb -D /var/lib/pgsql/data <span class="es1">\<br /> </span>--locale en_US.UTF-8 --auth md5 --pwprompt"</span></div></div></pre><h2 id="_start_postgres">Start Postgres</h2> <p>Now that a cluster exists, start the Postgres server using either the command provided to you in the output of <code>initdb</code> or with systemd:</p> <pre class="highlight"> <span class="geshifilter"><code class="bash geshifilter-bash"><span class="co4">$ </span><span class="kw2">sudo</span> systemctl start postgresql</code></span></pre><h2 id="_creating_a_database_user">Creating a database user</h2> <p>To create a Postgres user, use the <code>createuser</code> command. The <code>postgres</code> user is the superuser of the Postgres install,</p> <pre class="highlight"> <div class="geshifilter"><div class="bash geshifilter-bash">$ <span class="kw2">sudo</span> <span class="re5">-u</span> postgres createuser <span class="re5">--interactive</span> <span class="re5">--password</span> bogus<br /> Shall the new role be a superuser? <span class="br0">(</span>y<span class="sy0">/</span>n<span class="br0">)</span> n<br /> Shall the new role be allowed to create databases? <span class="br0">(</span>y<span class="sy0">/</span>n<span class="br0">)</span> y<br /> Shall the new role be allowed to create <span class="kw2">more</span> new roles? <span class="br0">(</span>y<span class="sy0">/</span>n<span class="br0">)</span> n<br /> Password:</div></div></pre><h2 id="_create_a_database">Create a database</h2> <p>To create a new database, use the <code>createdb</code> command. In this example, I create the database <code>exampledb</code> and assign ownership of it to the user <code>bogus</code>:</p> <pre class="highlight"> <span class="geshifilter"><code class="bash geshifilter-bash"><span class="co4">$ </span>createdb exampledb <span class="re5">--owner</span> bogus</code></span></pre><h2 id="_interacting_with_postgresql">Interacting with PostgreSQL</h2> <p>You can interact with a PostgreSQL database using the <code>psql</code> command. This command provides an interactive shell so you can view and update your databases. To connect to a database, specify the user and database you want to use:</p> <pre class="highlight"> <div class="geshifilter"><div class="bash geshifilter-bash">$ psql <span class="re5">--user</span> bogus exampledb<br /> psql <span class="br0">(</span>XX.Y<span class="br0">)</span><br /> Type <span class="st0">"help"</span> <span class="kw1">for</span> help.<br /> <br /> <span class="re2">exampledb</span>=<span class="sy0">&gt;</span></div></div></pre><h2 id="_create_a_table">Create a table</h2> <p>Databases contain tables, which can be visualized as a spreadsheet. There's a series of rows (called <em>records</em> in a database) and columns. The intersection of a row and a column is called a <em>field</em>.</p> <p>The Structured Query Language (SQL) is named after what it provides: A method to inquire about the contents of a database in a predictable and consistent syntax to receive useful results.</p> <p>Currently, your database is empty, devoid of any tables. You can create a table with the <code>CREATE</code> query. It's useful to combine this with the <code>IF NOT EXISTS</code> statement, which prevents PostgreSQL from clobbering an existing table.</p> <p>Before you createa table, think about what kind of data (the "data type" in SQL terminology) you anticipate the table to contain. In this example, I create a table with one column for a unique identifier and one column for some arbitrary text up to nine characters.</p> <pre class="highlight"> <div class="geshifilter"><div class="sql geshifilter-sql">exampledb<span class="sy0">=&gt;</span> <span class="kw1">CREATE</span> <span class="kw1">TABLE</span> <span class="kw1">IF</span> <span class="kw1">NOT</span> <span class="kw1">EXISTS</span> my_sample_table<span class="br0">(</span><br /> exampledb<span class="br0">(</span><span class="sy0">&gt;</span> id SERIAL<span class="sy0">,</span><br /> exampledb<span class="br0">(</span><span class="sy0">&gt;</span> wordlist <span class="kw1">VARCHAR</span><span class="br0">(</span><span class="nu0">9</span><span class="br0">)</span> <span class="kw1">NOT</span> <span class="kw1">NULL</span><br /> <span class="br0">)</span>;</div></div></pre><p>The <code>SERIAL</code> keyword isn't actually a data type. It's <a href="https://www.postgresql.org/docs/current/datatype-numeric.html#DATATYPE-SERIAL">special notation in PostgreSQL</a> that creates an auto-incrementing integer field. The <code>VARCHAR</code> keyword is a data type indicating a variable number of characters within a limit. In this code, I've specified a maximum of 9 characters. There are lots of data types in PostgreSQL, so refer to the project documentation for a list of options.</p> <h2 id="_insert_data">Insert data</h2> <p>You can populate your new table with some sample data by using the <code>INSERT</code> SQL keyword:</p> <pre class="highlight"> <div class="geshifilter"><div class="sql geshifilter-sql">exampledb<span class="sy0">=&gt;</span> <span class="kw1">INSERT</span> <span class="kw1">INTO</span> my_sample_table <span class="br0">(</span>wordlist<span class="br0">)</span> <span class="kw1">VALUES</span> <span class="br0">(</span><span class="st0">'Alice'</span><span class="br0">)</span>;<br /> <span class="kw1">INSERT</span> <span class="nu0">0</span> <span class="nu0">1</span></div></div></pre><p>Your data entry fails, should you attempt to put more than 9 characters into the <code>wordlist</code> field:</p> <pre class="highlight"> <div class="geshifilter"><div class="sql geshifilter-sql">exampledb<span class="sy0">=&gt;</span> <span class="kw1">INSERT</span> <span class="kw1">INTO</span> my_sample_table <span class="br0">(</span>WORDLIST<span class="br0">)</span> <span class="kw1">VALUES</span> <span class="br0">(</span><span class="st0">'Alexandria'</span><span class="br0">)</span>;<br /> ERROR:  <span class="kw1">VALUE</span> too long <span class="kw1">FOR</span> <span class="kw1">TYPE</span> <span class="kw1">CHARACTER</span> <span class="kw1">VARYING</span><span class="br0">(</span><span class="nu0">9</span><span class="br0">)</span></div></div></pre><h2 id="_alter_a_table_or_column">Alter a table or column</h2> <p>When you need to change a field definition, you use the <code>ALTER</code> SQL keyword. For instance, should you decide that a nine character limit for <code>wordlist</code>, you can increase its allowance by setting its data type:</p> <pre class="highlight"> <div class="geshifilter"><div class="sql geshifilter-sql">exampledb<span class="sy0">=&gt;</span> <span class="kw1">ALTER</span> <span class="kw1">TABLE</span> my_sample_table<br /> <span class="kw1">ALTER</span> <span class="kw1">COLUMN</span> wordlist <span class="kw1">SET</span> <span class="kw1">DATA</span> <span class="kw1">TYPE</span> <span class="kw1">VARCHAR</span><span class="br0">(</span><span class="nu0">10</span><span class="br0">)</span>;<br /> <span class="kw1">ALTER</span> <span class="kw1">TABLE</span><br /> exampledb<span class="sy0">=&gt;</span> <span class="kw1">INSERT</span> <span class="kw1">INTO</span> my_sample_table <span class="br0">(</span>WORDLIST<span class="br0">)</span> <span class="kw1">VALUES</span> <span class="br0">(</span><span class="st0">'Alexandria'</span><span class="br0">)</span>;<br /> <span class="kw1">INSERT</span> <span class="nu0">0</span> <span class="nu0">1</span></div></div></pre><h2 id="_view_data_in_a_table">View data in a table</h2> <p>SQL is a query language, so you view the contents of a database through queries. Queries can be simple, or it can involve joining complex relationships between several different tables. To see everything in a table, use the <code>SELECT</code> keyword on <code>*</code> (an asterisk is a wildcard):</p> <pre class="highlight"> <div class="geshifilter"><div class="sql geshifilter-sql">exampledb<span class="sy0">=&gt;</span> <span class="kw1">SELECT</span> <span class="sy0">*</span> <span class="kw1">FROM</span> my_sample_table;<br />  id <span class="sy0">|</span>  wordlist<br /> \<span class="co1">----+------------</span><br />   <span class="nu0">1</span> <span class="sy0">|</span> Alice<br />   <span class="nu0">2</span> <span class="sy0">|</span> Bob<br />   <span class="nu0">3</span> <span class="sy0">|</span> Alexandria<br /> <span class="br0">(</span><span class="nu0">3</span> <span class="kw1">ROWS</span><span class="br0">)</span></div></div></pre><h2 id="_more_data">More data</h2> <p>PostgreSQL can handle a lot of data, but as with any database the key to success is how you design your database for storage and what you do with the data once you've got it stored. A relatively large public data set can be found on <a href="https://stats.oecd.org/Index.aspx?DataSetCode=LAND_COVER">OECD.org</a>, and using this you can try some advanced database techniques.</p> <p>First, download the data as comma-separated values (CSV) and save the file as <code>land-cover.csv</code> in your <code>Downloads</code> folder.</p> <p>Browse the data in a text editor or spreadsheet application to get an idea of what columns there are, and what kind of data each column contains. Look at the data carefully and keep an eye out for exceptions to an apparent rule. For instance, the <code>COU</code> column, containing a country code such as <code>AUS</code> for Australia and <code>GRC</code> for Greece, tends to be 3 characters until the oddity <code>BRIICS</code>.</p> <p>Once you understand the data you're working with, you can prepare a Postgres database:</p> <pre class="highlight"> <div class="geshifilter"><div class="bash geshifilter-bash">$ createdb landcoverdb <span class="re5">--owner</span> bogus<br /> $ psql <span class="re5">--user</span> bogus landcoverdb<br /> <span class="re2">landcoverdb</span>=<span class="sy0">&gt;</span> create table land_cover<span class="br0">(</span><br /> country_code varchar<span class="br0">(</span><span class="nu0">6</span><span class="br0">)</span>,<br /> country_name varchar<span class="br0">(</span><span class="nu0">76</span><span class="br0">)</span>,<br /> small_subnational_region_code varchar<span class="br0">(</span><span class="nu0">5</span><span class="br0">)</span>,<br /> small_subnational_region_name varchar<span class="br0">(</span><span class="nu0">14</span><span class="br0">)</span>,<br /> large_subnational_region_code varchar<span class="br0">(</span><span class="nu0">17</span><span class="br0">)</span>,<br /> large_subnational_region_name varchar<span class="br0">(</span><span class="nu0">44</span><span class="br0">)</span>,<br /> measure_code varchar<span class="br0">(</span><span class="nu0">13</span><span class="br0">)</span>,<br /> measure_name varchar<span class="br0">(</span><span class="nu0">29</span><span class="br0">)</span>,<br /> land_cover_class_code varchar<span class="br0">(</span><span class="nu0">17</span><span class="br0">)</span>,<br /> land_cover_class_name varchar<span class="br0">(</span><span class="nu0">19</span><span class="br0">)</span>,<br /> year_code integer,<br /> year_value integer,<br /> unit_code varchar<span class="br0">(</span><span class="nu0">3</span><span class="br0">)</span>,<br /> unit_name varchar<span class="br0">(</span><span class="nu0">17</span><span class="br0">)</span>,<br /> power_code integer,<br /> power_name varchar<span class="br0">(</span><span class="nu0">9</span><span class="br0">)</span>,<br /> reference_period_code varchar<span class="br0">(</span><span class="nu0">1</span><span class="br0">)</span>,<br /> reference_period_name varchar<span class="br0">(</span><span class="nu0">1</span><span class="br0">)</span>,<br /> value float<span class="br0">(</span><span class="nu0">8</span><span class="br0">)</span>,<br /> flag_codes varchar<span class="br0">(</span><span class="nu0">1</span><span class="br0">)</span>,<br /> flag_names varchar<span class="br0">(</span><span class="nu0">1</span><span class="br0">)</span><span class="br0">)</span>;</div></div></pre><h2 id="_importing_data">Importing data</h2> <p>Postgres can import CSV data directly using the special metacommand <code>\copy</code>:</p> <pre class="highlight"> <div class="geshifilter"><div class="bash geshifilter-bash"><span class="re2">landcoverdb</span>=<span class="sy0">&gt;</span> \copy land_cover from <span class="st_h">'~/land-cover.csv'</span> with csv header delimiter <span class="st_h">','</span><br /> COPY <span class="nu0">22113</span></div></div></pre><p>That's 22,113 records imported. Seems like a good start!</p> <p></p><div class="embedded-resource-list callout-float-right"> <div class="field field--name-title field--type-string field--label-hidden field__item">More on edge computing</div> <div class="field field--name-links field--type-link field--label-hidden field__items"> <div class="field__item"><a href="https://www.redhat.com/en/topics/edge-computing?intcmp=7016000000127cYAAQ">Understanding edge computing</a></div> <div class="field__item"><a href="https://opensource.com/article/21/2/linux-edge-computing?intcmp=7016000000127cYAAQ">Why Linux is critical to edge computing</a></div> <div class="field__item"><a href="https://opensource.com/downloads/kubernetes-raspberry-pi?intcmp=7016000000127cYAAQ">eBook: Running Kubernetes on your Raspberry Pi</a></div> <div class="field__item"><a href="https://www.redhat.com/en/engage/automated-enterprise-ebook-20171115?intcmp=701f2000000h4RcAAI">Download now: The automated enterprise eBook</a></div> <div class="field__item"><a href="https://opensource.com/downloads/home-automation-ebook?intcmp=701f2000000h4RcAAI">eBook: A practical guide to home automation using open source tools</a></div> <div class="field__item"><a href="https://www.redhat.com/en/engage/automation-at-edge-20220727?intcmp=701f2000000h4RcAAI">eBook: 7 examples of automation on the edge</a></div> <div class="field__item"><a href="https://opensource.com/tags/edge-computing?intcmp=701f2000000h4RcAAI">The latest on edge</a></div> </div> </div> <h2 id="_querying_data">Querying data</h2> <p>A broad <code>SELECT</code> statement to see all columns of all 22,113 records is possible, and Postgres very nicely pipes the output to a screen pager so you can scroll through the output at a leisurely pace. However, using advanced SQL you can get some useful views of what's otherwise some pretty raw data.</p> <pre class="highlight"> <div class="geshifilter"><div class="sql geshifilter-sql">landcoverdb<span class="sy0">=&gt;</span> <span class="kw1">SELECT</span><br />     lcm<span class="sy0">.</span>country_name<span class="sy0">,</span><br />     lcm<span class="sy0">.</span>year_value<span class="sy0">,</span><br />     <span class="kw1">SUM</span><span class="br0">(</span>lcm<span class="sy0">.</span>value<span class="br0">)</span> sum_value<br /> <span class="kw1">FROM</span> land_cover lcm<br /> <span class="kw1">JOIN</span> <span class="br0">(</span><br />     <span class="kw1">SELECT</span><br />         country_name<span class="sy0">,</span><br />         large_subnational_region_name<span class="sy0">,</span><br />         small_subnational_region_name<span class="sy0">,</span><br />         <span class="kw1">MAX</span><span class="br0">(</span>year_value<span class="br0">)</span> max_year_value<br />     <span class="kw1">FROM</span> land_cover<br />     <span class="kw1">GROUP</span> <span class="kw1">BY</span> country_name<span class="sy0">,</span><br />         large_subnational_region_name<span class="sy0">,</span><br />         small_subnational_region_name<br /> <span class="br0">)</span> <span class="kw1">AS</span> lcmyv<br /> <span class="kw1">ON</span><br />     lcm<span class="sy0">.</span>country_name <span class="sy0">=</span> lcmyv<span class="sy0">.</span>country_name <span class="kw1">AND</span><br />     lcm<span class="sy0">.</span>large_subnational_region_name <span class="sy0">=</span> lcmyv<span class="sy0">.</span>large_subnational_region_name <span class="kw1">AND</span><br />     lcm<span class="sy0">.</span>small_subnational_region_name <span class="sy0">=</span> lcmyv<span class="sy0">.</span>small_subnational_region_name <span class="kw1">AND</span><br />     lcm<span class="sy0">.</span>year_value <span class="sy0">=</span> lcmyv<span class="sy0">.</span>max_year_value<br /> <span class="kw1">GROUP</span> <span class="kw1">BY</span> lcm<span class="sy0">.</span>country_name<span class="sy0">,</span><br />     lcm<span class="sy0">.</span>large_subnational_region_name<span class="sy0">,</span><br />     lcm<span class="sy0">.</span>small_subnational_region_name<span class="sy0">,</span><br />     lcm<span class="sy0">.</span>year_value<br /> <span class="kw1">ORDER</span> <span class="kw1">BY</span> country_name<span class="sy0">,</span><br />     year_value;</div></div></pre><p>Here's some sample output:</p> <pre class="highlight"> <div class="geshifilter"><div class="text geshifilter-text">\---------------+------------+------------<br />  Afghanistan    |       2019 |  743.48425<br />  Albania        |       2019 |  128.82532<br />  Algeria        |       2019 |  2417.3281<br />  American Samoa |       2019 |   100.2007<br />  Andorra        |       2019 |  100.45613<br />  Angola         |       2019 |  1354.2192<br />  Anguilla       |       2019 | 100.078514<br />  Antarctica     |       2019 |  12561.907<br /> [...]</div></div></pre><p>SQL is a rich langauge, and so it's beyond the scope of this article. Read through the SQL code and see if you can modify it to provide a different set of data.</p> <h2 id="_open_database">Open database</h2> <p>PostgreSQL is one of the great open source databases. With it, you can design repositories for structured data, and then use SQL to view it in different ways so you can gain fresh perspectives on that data. Postgres integrates with many languages, including Python, Lua, Groovy, Java, and more, so regardless of your toolset, you can probably make use of this excellent database.</p> </div> <div class="clearfix text-formatted field field--name-field-article-subhead field--type-text-long field--label-hidden field__item"><p>Postgres is one of the most flexible databases available, and it&#039;s open source.</p> </div> <div class="field field--name-field-lead-image field--type-entity-reference field--label-hidden field__item"><article class="media media--type-image media--view-mode-caption"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="/sites/default/files/lead-images/lenovo-thinkpad-laptop-concentration-focus-windows-office.png" width="799" height="447" alt="Business woman on laptop sitting in front of window" title="Woman using laptop concentrating" /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p>Image by Mapbox Uncharted ERG, <a href="https://creativecommons.org/licenses/by/3.0/us/" rel="ugc" target="_blank">CC-BY 3.0 US</a></p> </div> </article> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-hidden field__items"> <div class="field__item"><a href="/tags/databases" hreflang="en">Databases</a></div> <div class="field__item"><a href="/tags/edge-computing" hreflang="en">Edge computing</a></div> </div> <div class="field field--name-field-listicle-title field--type-string field--label-hidden field__item">What to read next</div> <div class="field field--name-field-listicles field--type-entity-reference field--label-hidden field__items"> <div class="field__item"><a href="/article/20/10/mariadb-mysql-linux" hreflang="und">Install MariaDB or MySQL on Linux</a></div> <div class="field__item"><a href="/article/20/2/postgresql-commands" hreflang="und">3 ways to use PostgreSQL commands</a></div> </div> <div class="field field--name-field-default-license field--type-list-string field--label-hidden field__item"><a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"> <img alt="Creative Commons License" src="/themes/osdc/assets/img/cc-by-sa-4.png" title="This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License." /></a>This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.</div> <article class="profile"> <a href="/points-and-badges" class="profile__points">6870 points</a> <div class="profile__container"> <span class="user-attributes">(Correspondent) </span> <div class="field field--name-field-user-picture field--type-entity-reference field--label-hidden field__item"><article class="media media--type-image media--view-mode-no-captions"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="/sites/default/files/clh_portrait2.jpg" width="344" height="384" alt="Chris Hermansen portrait Temuco Chile" /> </div> </article> </div> <div class="field field--name-field-location field--type-string field--label-hidden field__item">Vancouver, Canada</div> <div class="clearfix text-formatted field field--name-field-user-about field--type-text-with-summary field--label-hidden field__item"><p>Seldom without a computer of some sort since graduating from the University of British Columbia in 1978, I have been a full-time Linux user since 2005, a full-time Solaris and SunOS user from 1986 through 2005, and UNIX System V user before that.</p> <p>On the technical side of things, I have spent a great deal of my career as a consultant, doing data analysis and visualization; especially spatial data analysis. I have a substantial amount of related programming experience, using C, awk, Java, Python, PostgreSQL, PostGIS and lately Groovy. I'm looking at Julia with great interest. I have also built a few desktop and web-based applications, primarily in Java and lately in Grails with lots of JavaScript on the front end and PostgreSQL as my database of choice.</p> <p>Aside from that, I spend a considerable amount of time writing proposals, technical reports and - of course - stuff on <a href="https://www.opensource.com" rel="ugc">https://www.opensource.com</a>.</p> </div> <div class="field field--name-field-user-attributes field--type-entity-reference field--label-above"> <div class="field__label">User Attributes</div> <div class="field__items"> <div class="field__item"><a href="/taxonomy/term/16038" hreflang="en">Correspondent</a></div> </div> </div> </div> <div class="profile__badges"> <a href="/points-and-badges" class="field field--name-name field--type-string field--label-above profile__badges__name profile__badges__color-">Open Sourcerer</a> <a href="/points-and-badges" class="field field--name-name field--type-string field--label-above profile__badges__name profile__badges__color-">People&#039;s Choice Award</a> <a href="/points-and-badges" class="field field--name-name field--type-string field--label-above profile__badges__name profile__badges__color-">100+ Contributions Club</a> <a href="/points-and-badges" class="field field--name-name field--type-string field--label-above profile__badges__name profile__badges__color-">Emerging Contributor Award 2016</a> <a href="/points-and-badges" class="field field--name-name field--type-string field--label-above profile__badges__name profile__badges__color-">Author</a> <a href="/points-and-badges" class="field field--name-name field--type-string field--label-above profile__badges__name profile__badges__color-">Comment Gardener</a> <a href="/correspondent-program" class="field field--name-name field--type-string field--label-above profile__badges__name profile__badges__color-">Correspondent</a> <a href="/points-and-badges" class="field field--name-name field--type-string field--label-above profile__badges__name profile__badges__color-">Columnist</a> <a href="/resources/contributor-club" class="field field--name-name field--type-string field--label-above profile__badges__name profile__badges__color-">Contributor Club</a> </div> </article> <section id='comments' class="field field--name-field-comments field--type-comment field--label-hidden comment-wrapper"> <div class="comments__count"> <h2>2 Comments</h2> <div class="login"><a href="/user/register?absolute=1">Register</a> or <a href="/user/login?current=/rss.xml&amp;absolute=1">Login</a> to post a comment.</div> </div> <article data-comment-user-id="30666" id="comment-219878" class="comment js-comment"> <mark class="hidden" data-comment-timestamp="1664024673"></mark> <footer class="comment__meta"> <article class="media media--type-image media--view-mode-byline"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="/sites/default/files/styles/medium/public/20150529_gregp.jpg?itok=pjjrMm9R" width="189" height="220" alt="Greg Pittman" class="image-style-medium" /> </div> </article> <div> <div class="comment__submitted"> <a href="/users/greg-p">Greg Pittman</a> | September 24, 2022 <div class="rate-widget thumbsup"><form class="comment-content-219878-updown-rate-widget-0 vote-form" id="rate-widget-base-form--2" data-drupal-selector="comment-content-219878-updown-rate-widget-0" action="/feed" method="post" accept-charset="UTF-8"> <div class="rate-widget-thumbs-up"> <div class="thumbsup-rating-wrapper rate-disabled" can-edit="false"><div class="form-item js-form-item form-type-radio js-form-type-radio form-item-value js-form-item-value form-disabled"> <label class="form-item js-form-item form-type-radio js-form-type-radio form-item-value js-form-item-value form-disabled rating-label thumbsup-rating-label thumbsup-rating-label-up vote-pending rate-thumbs-up-btn-up"><input twig-suggestion="rating-input" class="rating-input thumbsup-rating-input form-radio" data-drupal-selector="edit-value-1-2" disabled="disabled" type="radio" id="edit-value-1--2" name="value" value="1" /> </label> </div> </div><input autocomplete="off" data-drupal-selector="form-fp1pvkmzyhiw2p2yodxpnistaarehagx-m-jpw-gkjq" type="hidden" name="form_build_id" value="form-fp1pVKMZyhiw2p2YoDXpnISTAaReHAgX_M_jPW-gKjQ" /> <input data-drupal-selector="edit-comment-content-219878-updown-rate-widget-0" type="hidden" name="form_id" value="comment_content_219878_updown_rate_widget_0" /> <input class="thumbsup-rating-submit button button--primary js-form-submit form-submit" data-drupal-selector="edit-submit" type="submit" id="edit-submit--4" name="op" value="Save" /> <div class="vote-result js-form-wrapper form-wrapper" data-drupal-selector="edit-result" id="edit-result--2"> <div title="Register or Login to like."><a href="/user/register">Register</a> or <a href="/user/login?current=/rss.xml">Login</a> to like</div> </div> <div class="rate-score"></div> </div> </form> </div> </div> <div class="content"> <div class="clearfix text-formatted field field--name-comment-body field--type-text-long field--label-hidden field__item"><p>After you do<br /> sudo systemctl start postgresql.service<br /> you can do<br /> sudo systemctl enable postgresql.service<br /> so that the postgresql server starts each time you boot your machine.<br /> Incidentally, my system needs the .service after postgresql.</p> </div> <drupal-render-placeholder callback="flag.link_builder:build" arguments="0=comment&amp;1=219878&amp;2=report_as_spam" token="loWSl-EDuqnp9-217XTOOkPmb1yaX3-qu3x3BRpEvx0"></drupal-render-placeholder><drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=219878&amp;1=full&amp;2=en&amp;3=" token="0Xmj527en3z9OxGu8kQ5R9gQOWZEVMt6Psj3uhSIInQ"></drupal-render-placeholder> </div> </div> </footer> </article> <article data-comment-user-id="240616" id="comment-219881" class="comment js-comment"> <mark class="hidden" data-comment-timestamp="1664137548"></mark> <footer class="comment__meta"> <article class="media media--type-image media--view-mode-byline"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="/sites/default/files/styles/medium/public/pictures/photo1.png?itok=RdWZdeOb" width="172" height="220" class="image-style-medium" /> </div> </article> <div> <div class="comment__submitted"> <a href="/users/hossow">Heiko Staudt</a> | September 25, 2022 <div class="rate-widget thumbsup"><form class="comment-content-219881-updown-rate-widget-0 vote-form" id="rate-widget-base-form--3" data-drupal-selector="comment-content-219881-updown-rate-widget-0" action="/feed" method="post" accept-charset="UTF-8"> <div class="rate-widget-thumbs-up"> <div class="thumbsup-rating-wrapper rate-disabled" can-edit="false"><div class="form-item js-form-item form-type-radio js-form-type-radio form-item-value js-form-item-value form-disabled"> <label class="form-item js-form-item form-type-radio js-form-type-radio form-item-value js-form-item-value form-disabled rating-label thumbsup-rating-label thumbsup-rating-label-up vote-pending rate-thumbs-up-btn-up"><input twig-suggestion="rating-input" class="rating-input thumbsup-rating-input form-radio" data-drupal-selector="edit-value-1-3" disabled="disabled" type="radio" id="edit-value-1--3" name="value" value="1" /> </label> </div> </div><input autocomplete="off" data-drupal-selector="form-kqglcemb8ivxidmklcvcb7u1v7ctds-1-vab202cbnu" type="hidden" name="form_build_id" value="form-KQgLcEmb8iVXidMKLCVCB7u1V7ctdS_1_vaB202CbnU" /> <input data-drupal-selector="edit-comment-content-219881-updown-rate-widget-0" type="hidden" name="form_id" value="comment_content_219881_updown_rate_widget_0" /> <input class="thumbsup-rating-submit button button--primary js-form-submit form-submit" data-drupal-selector="edit-submit" type="submit" id="edit-submit--6" name="op" value="Save" /> <div class="vote-result js-form-wrapper form-wrapper" data-drupal-selector="edit-result" id="edit-result--3"> <div title="Register or Login to like."><a href="/user/register">Register</a> or <a href="/user/login?current=/rss.xml">Login</a> to like</div> </div> <div class="rate-score"></div> </div> </form> </div> </div> <div class="content"> <div class="clearfix text-formatted field field--name-comment-body field--type-text-long field--label-hidden field__item"><p>I use postgreSQL for some years now.<br /> I switched because it has own data types for IPv4 and IPv6 addresses like <tt>inet</tt> and <tt>cidr</tt> . So i can query a table for IPs which are in a given network. Oh, and not to forget the <tt>macaddr</tt>.<br /> My personal "dream-team" is postgreSQL with Flask or Django.</p> </div> <drupal-render-placeholder callback="flag.link_builder:build" arguments="0=comment&amp;1=219881&amp;2=report_as_spam" token="fMcynh04pE_1WJXOIP8W9-KOdIjAZMJmrx6tTv7ivqM"></drupal-render-placeholder><drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=219881&amp;1=full&amp;2=en&amp;3=" token="Zi9M9wZAel8nwoftN6Px3f7KrOGgNJY-lngkoPBF7_o"></drupal-render-placeholder> </div> </div> </footer> </article> </section> Sat, 24 Sep 2022 07:00:00 +0000 Seth Kenlon 70199 at https://opensource.com https://opensource.com/article/22/9/drop-your-database-for-postgresql#comments How to build a dynamic distributed database with DistSQL https://opensource.com/article/22/9/dynamic-distributed-database-distsql <span class="field field--name-title field--type-string field--label-hidden">How to build a dynamic distributed database with DistSQL</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><a title="View user profile." href="/users/raigor" class="username">Raigor Jiang</a></span> <span class="field field--name-created field--type-created field--label-hidden">Fri, 09/23/2022 - 03:00</span> <div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>Distributed databases are common for many reasons. They increase reliability, redundancy, and performance. Apache ShardingSphere is an open source framework that enables you to transform any database into a distributed database. Since the release of ShardingSphere 5.0.0, DistSQL (Distributed SQL) has provided dynamic management for the ShardingSphere ecosystem.</p> <p>In this article, I demonstrate a data sharding scenario in which DistSQL's flexibility allows you to create a distributed database. At the same time, I show some syntax sugar to simplify operating procedures, allowing your potential users to choose their preferred syntax.</p> <p>A series of DistSQL statements are run through practical cases to give you a complete set of practical DistSQL sharding management methods, which create and maintain distributed databases through dynamic management.</p> <article class="align-center media media--type-image media--view-mode-default"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="https://opensource.com/sites/default/files/2022-09/1sharding.png" width="852" height="550" alt="Diagram of database sharding management options" /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p>(Jiang Longtao, CC BY-SA 4.0)</p> </div> </article> <h2>What is sharding?</h2> <p>In database terminology, <em>sharding</em> is the process of partitioning a table into separate entities. While the table data is directly related, it often exists on different physical database nodes or, at the very least, within separate logical partitions.</p> <h2>Practical case example</h2> <p>To follow along with this example, you must have these components in place, either in your lab or in your mind as you read this article:</p> <ul> <li>Two sharding tables: <strong>t_order</strong> and <strong>t_order_item.</strong></li> <li>For both tables, database shards are carried out with the <strong>user_id</strong> field, and table shards with the <strong>order_id</strong> field.</li> <li>The number of shards is two databases times three tables.</li> </ul> <article class="align-center media media--type-image media--view-mode-default"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="https://opensource.com/sites/default/files/2022-09/2shardingsphere.png" width="852" height="570" alt="Apache ShardingSphere databases" /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p>(Jiang Longtao, CC BY-SA 4.0)</p> </div> </article> <h2>Set up the environment</h2> <p>1. Prepare a database (MySQL, MariaDB, PostgreSQL, or openGauss) instance for access. Create two new databases: <strong>demo_ds_0</strong> and <strong>demo_ds_1</strong>.</p> <p>2. Deploy <a href="https://shardingsphere.apache.org/document/5.1.2/en/overview/" target="_blank">Apache ShardingSphere-Proxy 5.1.2</a> and <a href="https://zookeeper.apache.org/" target="_blank">Apache ZooKeeper</a>. ZooKeeper acts as a governance center and stores ShardingSphere metadata information.</p> <p>3. Configure <code>server.yaml</code> in the Proxy conf directory as follows:</p> <pre> <div class="geshifilter"><div class="yaml geshifilter-yaml"><span class="co4">mode</span>:<span class="co3"><br />   type</span><span class="sy2">: </span>Cluster<span class="co4"><br />   repository</span>:<span class="co3"><br />     type</span><span class="sy2">: </span>ZooKeeper<span class="co4"><br />     props</span>:<span class="co3"><br />       namespace</span><span class="sy2">: </span>governance_ds<span class="co3"><br />       server-lists</span><span class="sy2">: </span>localhost:2181 <span class="co1">#ZooKeeper address</span><span class="co3"><br />       retryIntervalMilliseconds</span><span class="sy2">: </span>500<span class="co3"><br />       timeToLiveSeconds</span><span class="sy2">: </span>60<span class="co3"><br />       maxRetries</span><span class="sy2">: </span>3<span class="co3"><br />       operationTimeoutMilliseconds</span><span class="sy2">: </span>500<span class="co3"><br />   overwrite</span><span class="sy2">: </span>falserules<span class="sy2">:<br /> </span>  - !AUTHORITY<span class="co4"><br />     users</span><span class="sy2">:<br /> </span>      - root@<span class="co2">%:root</span></div></div></pre><p>4. Start ShardingSphere-Proxy and connect it to Proxy using a client, for example:</p> <pre> <span class="geshifilter"><code class="bash geshifilter-bash"><span class="co4">$ </span>mysql <span class="re5">-h</span> 127.0.0.1 <span class="re5">-P</span> <span class="nu0">3307</span> <span class="re5">-u</span> root <span class="re5">-p</span></code></span></pre><p>5. Create a distributed database:</p> <pre> <span class="geshifilter"><code class="text geshifilter-text">CREATE DATABASE sharding_db;USE sharding_db;</code></span></pre><h3><strong>Add storage resources</strong></h3> <p>Next, add storage resources corresponding to the database:</p> <pre> <div class="geshifilter"><div class="text geshifilter-text">ADD RESOURCE ds_0 (<br />     HOST=127.0.0.1,<br />     PORT=3306,<br />     DB=demo_ds_0,<br />     USER=root,<br />     PASSWORD=123456<br /> ), ds_1(<br />     HOST=127.0.0.1,<br />     PORT=3306,<br />     DB=demo_ds_1,<br />     USER=root,<br />     PASSWORD=123456<br /> );</div></div></pre><p>View the storage resources:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> SHOW DATABASE RESOURCES\G;<br /> <span class="sy0">********</span> <span class="nu0">1</span>. row <span class="sy0">***************************</span><br />          name: ds_1<br />          type: MySQL<br />          host: 127.0.0.1<br />          port: <span class="nu0">3306</span><br />            db: demo_ds_1<br />           <span class="re5">--</span> Omit partial attributes<br /> <span class="sy0">********</span> <span class="nu0">2</span>. row <span class="sy0">***************************</span><br />          name: ds_0<br />          type: MySQL<br />          host: 127.0.0.1<br />          port: <span class="nu0">3306</span><br />            db: demo_ds_0<br />           <span class="re5">--</span> Omit partial attributes</div></div></pre><p>Adding the optional <code>\G</code> switch to the query statement makes the output format easy to read.</p> <p></p><div class="embedded-resource-list callout-float-right"> <div class="field field--name-title field--type-string field--label-hidden field__item">More on data science</div> <div class="field field--name-links field--type-link field--label-hidden field__items"> <div class="field__item"><a href="https://opensource.com/resources/data-science?intcmp=7013a000002CxqkAAC">What is data science?</a></div> <div class="field__item"><a href="https://opensource.com/resources/python?intcmp=7013a000002CxqkAAC">What is Python?</a></div> <div class="field__item"><a href="https://opensource.com/article/17/9/data-scientist?intcmp=7013a000002CxqkAAC">How to become a data scientist</a></div> <div class="field__item"><a href="https://enterprisersproject.com/article/2022/9/data-scientist-day-life?intcmp=7013a000002CxqkAAC">Data scientist: A day in the life</a></div> <div class="field__item"><a href="https://opensource.com/resources/big-data?intcmp=7013a000002CxqkAAC">What is big data?</a></div> <div class="field__item"><a href="https://www.redhat.com/en/resources/data-intensive-applications-hybrid-cloud-blueprint-detail?intcmp=7013a000002CxqkAAC">Whitepaper: Data-intensive intelligent applications in a hybrid cloud blueprint</a></div> <div class="field__item"><a href="https://opensource.com/downloads/mariadb-mysql-cheat-sheet?intcmp=7013a000002CxqkAAC">MariaDB and MySQL cheat sheet</a></div> <div class="field__item"><a href="https://opensource.com/tags/data-science?intcmp=7013a000002CxqkAAC">Latest data science articles</a></div> </div> </div> <h2>Create sharding rules</h2> <p>ShardingSphere's sharding rules support regular sharding and automatic sharding. Both sharding methods have the same effect. The difference is that the configuration of automatic sharding is more concise, while regular sharding is more flexible and independent.</p> <p>Refer to the following links for more details on automatic sharding:</p> <ul> <li><a href="https://medium.com/nerd-for-tech/intro-to-distsql-an-open-source-more-powerful-sql-bada4099211?source=your_stories_page-------------------------------------" target="_blank">Intro to DistSQL-An Open Source and More Powerful SQL</a></li> <li><a href="https://medium.com/geekculture/autotable-your-butler-like-sharding-configuration-tool-9a45dbb7e285" target="_blank">AutoTable: Your Butler-Like Sharding Configuration Tool</a></li> </ul> <p>Next, it's time to adopt regular sharding and use the <strong>INLINE</strong> expression algorithm to implement the sharding scenarios described in the requirements.</p> <h2>Primary key generator</h2> <p>The primary key generator creates a secure and unique primary key for a data table in a distributed scenario. For details, refer to the document <a href="https://shardingsphere.apache.org/document/current/en/features/sharding/concept/#distributed-primary-key" target="_blank">Distributed Primary Key</a><em>.</em></p> <p>1. Create a primary key generator:</p> <pre> <div class="geshifilter"><div class="text geshifilter-text">CREATE SHARDING KEY GENERATOR snowflake_key_generator (<br /> TYPE(NAME=SNOWFLAKE)<br /> );</div></div></pre><p>2. Query the primary key generator:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> SHOW SHARDING KEY GENERATORS;<br /> +-------------------------+-----------+-------+<br /> <span class="sy0">|</span> name                    <span class="sy0">|</span> <span class="kw3">type</span>      <span class="sy0">|</span> props <span class="sy0">|</span><br /> +-------------------------+-----------+-------+<br /> <span class="sy0">|</span> snowflake_key_generator <span class="sy0">|</span> snowflake <span class="sy0">|</span> <span class="br0">{</span><span class="br0">}</span>    <span class="sy0">|</span><br /> +-------------------------+-----------+-------+<br /> <span class="nu0">1</span> row <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.01</span> sec<span class="br0">)</span></div></div></pre><h2>Sharding algorithm</h2> <p>1. Create a database sharding algorithm used by <strong>t_order</strong> and <strong>t_order_item</strong> in common:</p> <pre> <div class="geshifilter"><div class="text geshifilter-text">-- Modulo 2 based on user_id in database sharding<br /> CREATE SHARDING ALGORITHM database_inline (<br /> TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="ds_${user_id % 2}"))<br /> );</div></div></pre><p>2. Create different table shards algorithms for <strong>t_order</strong> and <strong>t_order_item:</strong></p> <pre> <div class="geshifilter"><div class="text geshifilter-text">-- Modulo 3 based on order_id in table sharding<br /> CREATE SHARDING ALGORITHM t_order_inline (<br /> TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="t_order_${order_id % 3}"))<br /> );<br /> CREATE SHARDING ALGORITHM t_order_item_inline (<br /> TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="t_order_item_${order_id % 3}"))<br /> );</div></div></pre><p>3. Query the sharding algorithm:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> SHOW SHARDING ALGORITHMS;<br /> +---------------------+--------+---------------------------------------------------+<br /> <span class="sy0">|</span> name                <span class="sy0">|</span> <span class="kw3">type</span>   <span class="sy0">|</span> props                                             <span class="sy0">|</span><br /> +---------------------+--------+---------------------------------------------------+<br /> <span class="sy0">|</span> database_inline     <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=ds_<span class="co1">${user_id % 2}</span>            <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_inline      <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=t_order_<span class="co1">${order_id % 3}</span>      <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_item_inline <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=t_order_item_<span class="co1">${order_id % 3}</span> <span class="sy0">|</span><br /> +---------------------+--------+---------------------------------------------------+<br /> <span class="nu0">3</span> rows <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.00</span> sec<span class="br0">)</span></div></div></pre><h2>Create a default sharding strategy</h2> <p>The <a href="https://shardingsphere.apache.org/document/5.1.2/en/features/sharding/concept/sharding/" target="_blank">sharding strategy</a> consists of a sharding key and sharding algorithm, which in this case is <strong>databaseStrategy</strong> and <strong>tableStrategy</strong>. Because <strong>t_order</strong> and <strong>t_order_item</strong> have the same database sharding field and sharding algorithm, create a default strategy to be used by all shard tables with no sharding strategy configured.</p> <p>1. Create a default database sharding strategy:</p> <pre> <div class="geshifilter"><div class="text geshifilter-text">CREATE DEFAULT SHARDING DATABASE STRATEGY (<br /> TYPE=STANDARD,SHARDING_COLUMN=user_id,SHARDING_ALGORITHM=database_inline<br /> );</div></div></pre><p>2. Query default strategy:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> SHOW DEFAULT SHARDING STRATEGY\G;<br /> <span class="sy0">***************************</span> <span class="nu0">1</span>. row <span class="sy0">***************************</span><br />                     name: TABLE<br />                     type: NONE<br />          sharding_column:<br />  sharding_algorithm_name:<br />  sharding_algorithm_type:<br /> sharding_algorithm_props:<br /> <span class="sy0">***************************</span> <span class="nu0">2</span>. row <span class="sy0">***************************</span><br />                     name: DATABASE<br />                     type: STANDARD<br />          sharding_column: user_id<br />  sharding_algorithm_name: database_inline<br />  sharding_algorithm_type: inline<br /> sharding_algorithm_props: <span class="br0">{</span>algorithm-expression=ds_<span class="co1">${user_id % 2}</span><span class="br0">}</span><br /> <span class="nu0">2</span> rows <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.00</span> sec<span class="br0">)</span></div></div></pre><p>You have not configured the default table sharding strategy, so the default strategy of <strong>TABLE</strong> is <strong>NONE</strong>.</p> <h2>Set sharding rules</h2> <p>The primary key generator and sharding algorithm are both ready. Now you can create sharding rules. The method I demonstrate below is a little complicated and involves multiple steps. In the next section, I'll show you how to create sharding rules in just one step, but for now, witness how it's typically done.</p> <p>First, define <strong>t_order</strong>:</p> <pre> <div class="geshifilter"><div class="text geshifilter-text">CREATE SHARDING TABLE RULE t_order (<br /> DATANODES("ds_${0..1}.t_order_${0..2}"),<br /> TABLE_STRATEGY(TYPE=STANDARD,SHARDING_COLUMN=order_id,SHARDING_ALGORITHM=t_order_inline),<br /> KEY_GENERATE_STRATEGY(COLUMN=order_id,KEY_GENERATOR=snowflake_key_generator)<br /> );</div></div></pre><p>Here is an explanation of the values found above:</p> <ul> <li><strong>DATANODES</strong> specifies the data nodes of shard tables.</li> <li><strong>TABLE_STRATEGY</strong> specifies the table strategy, among which <strong>SHARDING_ALGORITHM</strong> uses created sharding algorithm <strong>t_order_inline</strong>.</li> <li><strong>KEY_GENERATE_STRATEGY</strong> specifies the primary key generation strategy of the table. Skip this configuration if primary key generation is not required.</li> </ul> <p>Next, define <strong>t_order_item</strong>:</p> <pre> <div class="geshifilter"><div class="text geshifilter-text">CREATE SHARDING TABLE RULE t_order_item (<br /> DATANODES("ds_${0..1}.t_order_item_${0..2}"),<br /> TABLE_STRATEGY(TYPE=STANDARD,SHARDING_COLUMN=order_id,SHARDING_ALGORITHM=t_order_item_inline),<br /> KEY_GENERATE_STRATEGY(COLUMN=order_item_id,KEY_GENERATOR=snowflake_key_generator)<br /> );</div></div></pre><p>Query the sharding rules to verify what you've created:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> SHOW SHARDING TABLE RULES\G;<br /> <span class="sy0">**************************</span> <span class="nu0">1</span>. row <span class="sy0">***************************</span><br />                            table: t_order<br />                actual_data_nodes: ds_<span class="co1">${0..1}</span>.t_order_<span class="co1">${0..2}</span><br />              actual_data_sources:<br />           database_strategy_type: STANDARD<br />         database_sharding_column: user_id<br /> database_sharding_algorithm_type: inline<br /> database_sharding_algorithm_props: algorithm-expression=ds_<span class="co1">${user_id % 2}</span><br />               table_strategy_type: STANDARD<br />             table_sharding_column: order_id<br />     table_sharding_algorithm_type: inline<br />    table_sharding_algorithm_props: algorithm-expression=t_order_<span class="co1">${order_id % 3}</span><br />               key_generate_column: order_id<br />                key_generator_type: snowflake<br />               key_generator_props:<br /> <span class="sy0">***************************</span> <span class="nu0">2</span>. row <span class="sy0">***************************</span><br />                             table: t_order_item<br />                 actual_data_nodes: ds_<span class="co1">${0..1}</span>.t_order_item_<span class="co1">${0..2}</span><br />               actual_data_sources:<br />            database_strategy_type: STANDARD<br />          database_sharding_column: user_id<br />  database_sharding_algorithm_type: inline<br /> database_sharding_algorithm_props: algorithm-expression=ds_<span class="co1">${user_id % 2}</span><br />               table_strategy_type: STANDARD<br />             table_sharding_column: order_id<br />     table_sharding_algorithm_type: inline<br />    table_sharding_algorithm_props: algorithm-expression=t_order_item_<span class="co1">${order_id % 3}</span><br />               key_generate_column: order_item_id<br />                key_generator_type: snowflake<br />               key_generator_props:<br /> <span class="nu0">2</span> rows <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.00</span> sec<span class="br0">)</span></div></div></pre><p>This looks right so far. You have now configured the sharding rules for <strong>t_order</strong> and <strong>t_order_item</strong>.</p> <p>You can skip the steps for creating the primary key generator, sharding algorithm, and default strategy, and complete the sharding rules in one step. Here's how to make it easier.</p> <h2>Sharding rule syntax</h2> <p>For instance, if you want to add a shard table called <strong>t_order_detail</strong>, you can create sharding rules as follows:</p> <pre> <div class="geshifilter"><div class="text geshifilter-text">CREATE SHARDING TABLE RULE t_order_detail (<br /> DATANODES("ds_${0..1}.t_order_detail_${0..1}"),<br /> DATABASE_STRATEGY(TYPE=STANDARD,SHARDING_COLUMN=user_id,SHARDING_ALGORITHM(TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="ds_${user_id % 2}")))),<br /> TABLE_STRATEGY(TYPE=STANDARD,SHARDING_COLUMN=order_id,SHARDING_ALGORITHM(TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="t_order_detail_${order_id % 3}")))),<br /> KEY_GENERATE_STRATEGY(COLUMN=detail_id,TYPE(NAME=snowflake))<br /> );</div></div></pre><p>This statement specifies a database sharding strategy, table strategy, and primary key generation strategy, but it doesn't use existing algorithms. The DistSQL engine automatically uses the input expression to create an algorithm for the sharding rules of <strong>t_order_detail</strong>.</p> <p>Now there's a primary key generator:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> SHOW SHARDING KEY GENERATORS;<br /> +--------------------------+-----------+-------+<br /> <span class="sy0">|</span> name                     <span class="sy0">|</span> <span class="kw3">type</span>      <span class="sy0">|</span> props <span class="sy0">|</span><br /> +--------------------------+-----------+-------+<br /> <span class="sy0">|</span> snowflake_key_generator  <span class="sy0">|</span> snowflake <span class="sy0">|</span> <span class="br0">{</span><span class="br0">}</span>    <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_detail_snowflake <span class="sy0">|</span> snowflake <span class="sy0">|</span> <span class="br0">{</span><span class="br0">}</span>    <span class="sy0">|</span><br /> +--------------------------+-----------+-------+<br /> <span class="nu0">2</span> rows <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.00</span> sec<span class="br0">)</span></div></div></pre><p>Display the sharding algorithm:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> SHOW SHARDING ALGORITHMS;<br /> +--------------------------------+--------+-----------------------------------------------------+<br /> <span class="sy0">|</span> name                           <span class="sy0">|</span> <span class="kw3">type</span>   <span class="sy0">|</span> props                                               <span class="sy0">|</span><br /> +--------------------------------+--------+-----------------------------------------------------+<br /> <span class="sy0">|</span> database_inline                <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=ds_<span class="co1">${user_id % 2}</span>              <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_inline                 <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=t_order_<span class="co1">${order_id % 3}</span>        <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_item_inline            <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=t_order_item_<span class="co1">${order_id % 3}</span>   <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_detail_database_inline <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=ds_<span class="co1">${user_id % 2}</span>              <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_detail_table_inline    <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=t_order_detail_<span class="co1">${order_id % 3}</span> <span class="sy0">|</span><br /> +--------------------------------+--------+-----------------------------------------------------+<br /> <span class="nu0">5</span> rows <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.00</span> sec<span class="br0">)</span></div></div></pre><p>And finally, the sharding rules:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> SHOW SHARDING TABLE RULES\G;<br /> <span class="sy0">***************************</span> <span class="nu0">1</span>. row <span class="sy0">***************************</span><br />                             table: t_order<br />                 actual_data_nodes: ds_<span class="co1">${0..1}</span>.t_order_<span class="co1">${0..2}</span><br />               actual_data_sources:<br />            database_strategy_type: STANDARD<br />          database_sharding_column: user_id<br />  database_sharding_algorithm_type: inline<br /> database_sharding_algorithm_props: algorithm-expression=ds_<span class="co1">${user_id % 2}</span><br />               table_strategy_type: STANDARD<br />             table_sharding_column: order_id<br />     table_sharding_algorithm_type: inline<br />    table_sharding_algorithm_props: algorithm-expression=t_order_<span class="co1">${order_id % 3}</span><br />               key_generate_column: order_id<br />                key_generator_type: snowflake<br />               key_generator_props:<br /> <span class="sy0">***************************</span> <span class="nu0">2</span>. row <span class="sy0">***************************</span><br />                             table: t_order_item<br />                 actual_data_nodes: ds_<span class="co1">${0..1}</span>.t_order_item_<span class="co1">${0..2}</span><br />               actual_data_sources:<br />            database_strategy_type: STANDARD<br />          database_sharding_column: user_id<br />  database_sharding_algorithm_type: inline<br /> database_sharding_algorithm_props: algorithm-expression=ds_<span class="co1">${user_id % 2}</span><br />               table_strategy_type: STANDARD<br />             table_sharding_column: order_id<br />     table_sharding_algorithm_type: inline<br />    table_sharding_algorithm_props: algorithm-expression=t_order_item_<span class="co1">${order_id % 3}</span><br />               key_generate_column: order_item_id<br />                key_generator_type: snowflake<br />               key_generator_props:<br /> <span class="sy0">***************************</span> <span class="nu0">3</span>. row <span class="sy0">***************************</span><br />                             table: t_order_detail<br />                 actual_data_nodes: ds_<span class="co1">${0..1}</span>.t_order_detail_<span class="co1">${0..1}</span><br />               actual_data_sources:<br />            database_strategy_type: STANDARD<br />          database_sharding_column: user_id<br />  database_sharding_algorithm_type: inline<br /> database_sharding_algorithm_props: algorithm-expression=ds_<span class="co1">${user_id % 2}</span><br />               table_strategy_type: STANDARD<br />             table_sharding_column: order_id<br />     table_sharding_algorithm_type: inline<br />    table_sharding_algorithm_props: algorithm-expression=t_order_detail_<span class="co1">${order_id % 3}</span><br />               key_generate_column: detail_id<br />                key_generator_type: snowflake<br />               key_generator_props:<br /> <span class="nu0">3</span> rows <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.01</span> sec<span class="br0">)</span></div></div></pre><p>In the <code>CREATE SHARDING TABLE RULE</code> statement, <strong>DATABASE_STRATEGY</strong>, <strong>TABLE_STRATEGY</strong>, and <strong>KEY_GENERATE_STRATEGY</strong> can reuse existing algorithms.</p> <p>Alternatively, they can be defined quickly through syntax. The difference is that additional algorithm objects are created.</p> <h2>Configuration and verification</h2> <p>Once you have created the configuration verification rules, you can verify them in the following ways.</p> <p>1. Check node distribution:</p> <p>DistSQL provides <code>SHOW SHARDING TABLE NODES</code> for checking node distribution, and users can quickly learn the distribution of shard tables:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> SHOW SHARDING TABLE NODES;<br /> +----------------+------------------------------------------------------------------------------------------------------------------------------+<br /> <span class="sy0">|</span> name           <span class="sy0">|</span> nodes                                                                                                                        <span class="sy0">|</span><br /> +----------------+------------------------------------------------------------------------------------------------------------------------------+<br /> <span class="sy0">|</span> t_order        <span class="sy0">|</span> ds_0.t_order_0, ds_0.t_order_1, ds_0.t_order_2, ds_1.t_order_0, ds_1.t_order_1, ds_1.t_order_2                               <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_item   <span class="sy0">|</span> ds_0.t_order_item_0, ds_0.t_order_item_1, ds_0.t_order_item_2, ds_1.t_order_item_0, ds_1.t_order_item_1, ds_1.t_order_item_2 <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_detail <span class="sy0">|</span> ds_0.t_order_detail_0, ds_0.t_order_detail_1, ds_1.t_order_detail_0, ds_1.t_order_detail_1                                   <span class="sy0">|</span><br /> +----------------+------------------------------------------------------------------------------------------------------------------------------+<br /> <span class="nu0">3</span> rows <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.01</span> sec<span class="br0">)</span><br /> <br /> mysql<span class="sy0">&gt;</span> SHOW SHARDING TABLE NODES t_order_item;<br /> +--------------+------------------------------------------------------------------------------------------------------------------------------+<br /> <span class="sy0">|</span> name         <span class="sy0">|</span> nodes                                                                                                                        <span class="sy0">|</span><br /> +--------------+------------------------------------------------------------------------------------------------------------------------------+<br /> <span class="sy0">|</span> t_order_item <span class="sy0">|</span> ds_0.t_order_item_0, ds_0.t_order_item_1, ds_0.t_order_item_2, ds_1.t_order_item_0, ds_1.t_order_item_1, ds_1.t_order_item_2 <span class="sy0">|</span><br /> +--------------+------------------------------------------------------------------------------------------------------------------------------+<br /> <span class="nu0">1</span> row <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.00</span> sec<span class="br0">)</span></div></div></pre><p>You can see that the node distribution of the shard table is consistent with what is described in the requirement.</p> <h2>SQL preview</h2> <p>Previewing SQL is also an easy way to verify configurations. Its syntax is <code>PREVIEW SQL</code>. First, make a query with no shard key, with all routes:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> PREVIEW SELECT <span class="sy0">*</span> FROM t_order;<br /> +------------------+---------------------------------------------------------------------------------------------+<br /> <span class="sy0">|</span> data_source_name <span class="sy0">|</span> actual_sql                                                                                  <span class="sy0">|</span><br /> +------------------+---------------------------------------------------------------------------------------------+<br /> <span class="sy0">|</span> ds_0             <span class="sy0">|</span> SELECT <span class="sy0">*</span> FROM t_order_0 UNION ALL SELECT <span class="sy0">*</span> FROM t_order_1 UNION ALL SELECT <span class="sy0">*</span> FROM t_order_2 <span class="sy0">|</span><br /> <span class="sy0">|</span> ds_1             <span class="sy0">|</span> SELECT <span class="sy0">*</span> FROM t_order_0 UNION ALL SELECT <span class="sy0">*</span> FROM t_order_1 UNION ALL SELECT <span class="sy0">*</span> FROM t_order_2 <span class="sy0">|</span><br /> +------------------+---------------------------------------------------------------------------------------------+<br /> <span class="nu0">2</span> rows <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.13</span> sec<span class="br0">)</span><br /> <br /> mysql<span class="sy0">&gt;</span> PREVIEW SELECT <span class="sy0">*</span> FROM t_order_item;<br /> +------------------+------------------------------------------------------------------------------------------------------------+<br /> <span class="sy0">|</span> data_source_name <span class="sy0">|</span> actual_sql                                                                                                 <span class="sy0">|</span><br /> +------------------+------------------------------------------------------------------------------------------------------------+<br /> <span class="sy0">|</span> ds_0             <span class="sy0">|</span> SELECT <span class="sy0">*</span> FROM t_order_item_0 UNION ALL SELECT <span class="sy0">*</span> FROM t_order_item_1 UNION ALL SELECT <span class="sy0">*</span> FROM t_order_item_2 <span class="sy0">|</span><br /> <span class="sy0">|</span> ds_1             <span class="sy0">|</span> SELECT <span class="sy0">*</span> FROM t_order_item_0 UNION ALL SELECT <span class="sy0">*</span> FROM t_order_item_1 UNION ALL SELECT <span class="sy0">*</span> FROM t_order_item_2 <span class="sy0">|</span><br /> +------------------+------------------------------------------------------------------------------------------------------------+<br /> <span class="nu0">2</span> rows <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.00</span> sec<span class="br0">)</span></div></div></pre><p>Now specify <strong>user_id</strong> in a query with a single database route:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> PREVIEW SELECT <span class="sy0">*</span> FROM t_order WHERE user_id = <span class="nu0">1</span>;<br /> +------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+<br /> <span class="sy0">|</span> data_source_name <span class="sy0">|</span> actual_sql                                                                                                                                        <span class="sy0">|</span><br /> +------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+<br /> <span class="sy0">|</span> ds_1             <span class="sy0">|</span> SELECT <span class="sy0">*</span> FROM t_order_0 WHERE user_id = <span class="nu0">1</span> UNION ALL SELECT <span class="sy0">*</span> FROM t_order_1 WHERE user_id = <span class="nu0">1</span> UNION ALL SELECT <span class="sy0">*</span> FROM t_order_2 WHERE user_id = <span class="nu0">1</span> <span class="sy0">|</span><br /> +------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+<br /> <span class="nu0">1</span> row <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.14</span> sec<span class="br0">)</span><br /> <br /> mysql<span class="sy0">&gt;</span> PREVIEW SELECT <span class="sy0">*</span> FROM t_order_item WHERE user_id = <span class="nu0">2</span>;<br /> +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br /> <span class="sy0">|</span> data_source_name <span class="sy0">|</span> actual_sql                                                                                                                                                       <span class="sy0">|</span><br /> +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br /> <span class="sy0">|</span> ds_0             <span class="sy0">|</span> SELECT <span class="sy0">*</span> FROM t_order_item_0 WHERE user_id = <span class="nu0">2</span> UNION ALL SELECT <span class="sy0">*</span> FROM t_order_item_1 WHERE user_id = <span class="nu0">2</span> UNION ALL SELECT <span class="sy0">*</span> FROM t_order_item_2 WHERE user_id = <span class="nu0">2</span> <span class="sy0">|</span><br /> +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br /> <span class="nu0">1</span> row <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.00</span> sec<span class="br0">)</span></div></div></pre><p>Specify <strong>user_id</strong> and <strong>order_id</strong> with a single table route:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> PREVIEW SELECT <span class="sy0">*</span> FROM t_order WHERE user_id = <span class="nu0">1</span> AND order_id = <span class="nu0">1</span>;<br /> +------------------+------------------------------------------------------------+<br /> <span class="sy0">|</span> data_source_name <span class="sy0">|</span> actual_sql                                                 <span class="sy0">|</span><br /> +------------------+------------------------------------------------------------+<br /> <span class="sy0">|</span> ds_1             <span class="sy0">|</span> SELECT <span class="sy0">*</span> FROM t_order_1 WHERE user_id = <span class="nu0">1</span> AND order_id = <span class="nu0">1</span> <span class="sy0">|</span><br /> +------------------+------------------------------------------------------------+<br /> <span class="nu0">1</span> row <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.04</span> sec<span class="br0">)</span><br /> <br /> mysql<span class="sy0">&gt;</span> PREVIEW SELECT <span class="sy0">*</span> FROM t_order_item WHERE user_id = <span class="nu0">2</span> AND order_id = <span class="nu0">5</span>;<br /> +------------------+-----------------------------------------------------------------+<br /> <span class="sy0">|</span> data_source_name <span class="sy0">|</span> actual_sql                                                      <span class="sy0">|</span><br /> +------------------+-----------------------------------------------------------------+<br /> <span class="sy0">|</span> ds_0             <span class="sy0">|</span> SELECT <span class="sy0">*</span> FROM t_order_item_2 WHERE user_id = <span class="nu0">2</span> AND order_id = <span class="nu0">5</span> <span class="sy0">|</span><br /> +------------------+-----------------------------------------------------------------+<br /> <span class="nu0">1</span> row <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.01</span> sec<span class="br0">)</span></div></div></pre><p>Single-table routes scan the fewest shard tables and offer the highest efficiency.</p> <h2>Query unused resources</h2> <p>During system maintenance, algorithms or storage resources that are no longer in use may need to be released, or resources that need to be released may have been referenced and cannot be deleted. DistSQL's <code>SHOW UNUSED RESOURCES</code> command can solve these problems:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> ADD RESOURCE ds_2 <span class="br0">(</span><br />     -<span class="sy0">&gt;</span>     <span class="re2">HOST</span>=127.0.0.1,<br />     -<span class="sy0">&gt;</span>     <span class="re2">PORT</span>=<span class="nu0">3306</span>,<br />     -<span class="sy0">&gt;</span>     <span class="re2">DB</span>=demo_ds_2,<br />     -<span class="sy0">&gt;</span>     <span class="re2">USER</span>=root,<br />     -<span class="sy0">&gt;</span>     <span class="re2">PASSWORD</span>=<span class="nu0">123456</span><br />     -<span class="sy0">&gt;</span> <span class="br0">)</span>;<br /> Query OK, <span class="nu0">0</span> rows affected <span class="br0">(</span><span class="nu0">0.07</span> sec<span class="br0">)</span><br /> <br /> mysql<span class="sy0">&gt;</span> SHOW UNUSED RESOURCES\G;<br /> <span class="sy0">***************************</span> <span class="nu0">1</span>. row <span class="sy0">***************************</span><br />                            name: ds_2<br />                            type: MySQL<br />                            host: 127.0.0.1<br />                            port: <span class="nu0">3306</span><br />                              db: demo_ds_2<br /> connection_timeout_milliseconds: <span class="nu0">30000</span><br />       idle_timeout_milliseconds: <span class="nu0">60000</span><br />       max_lifetime_milliseconds: <span class="nu0">2100000</span><br />                   max_pool_size: <span class="nu0">50</span><br />                   min_pool_size: <span class="nu0">1</span><br />                       read_only: <span class="kw2">false</span><br />                other_attributes: <span class="br0">{</span><span class="st0">"dataSourceProperties"</span>:<span class="br0">{</span><span class="st0">"cacheServerConfiguration"</span>:<span class="st0">"true"</span>,<span class="st0">"elideSetAutoCommits"</span>:<span class="st0">"true"</span>,<span class="st0">"useServerPrepStmts"</span>:<span class="st0">"true"</span>,<span class="st0">"cachePrepStmts"</span>:<span class="st0">"true"</span>,<span class="st0">"useSSL"</span>:<span class="st0">"false"</span>,<span class="st0">"rewriteBatchedStatements"</span>:<span class="st0">"true"</span>,<span class="st0">"cacheResultSetMetadata"</span>:<span class="st0">"false"</span>,<span class="st0">"useLocalSessionState"</span>:<span class="st0">"true"</span>,<span class="st0">"maintainTimeStats"</span>:<span class="st0">"false"</span>,<span class="st0">"prepStmtCacheSize"</span>:<span class="st0">"200000"</span>,<span class="st0">"tinyInt1isBit"</span>:<span class="st0">"false"</span>,<span class="st0">"prepStmtCacheSqlLimit"</span>:<span class="st0">"2048"</span>,<span class="st0">"serverTimezone"</span>:<span class="st0">"UTC"</span>,<span class="st0">"netTimeoutForStreamingResults"</span>:<span class="st0">"0"</span>,<span class="st0">"zeroDateTimeBehavior"</span>:<span class="st0">"round"</span><span class="br0">}</span>,<span class="st0">"healthCheckProperties"</span>:<span class="br0">{</span><span class="br0">}</span>,<span class="st0">"initializationFailTimeout"</span>:<span class="nu0">1</span>,<span class="st0">"validationTimeout"</span>:<span class="nu0">5000</span>,<span class="st0">"leakDetectionThreshold"</span>:<span class="nu0">0</span>,<span class="st0">"poolName"</span>:<span class="st0">"HikariPool-8"</span>,<span class="st0">"registerMbeans"</span>:false,<span class="st0">"allowPoolSuspension"</span>:false,<span class="st0">"autoCommit"</span>:true,<span class="st0">"isolateInternalQueries"</span>:false<span class="br0">}</span><br /> <span class="nu0">1</span> row <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.03</span> sec<span class="br0">)</span></div></div></pre><h3>Query unused primary key generator</h3> <p>DistSQL can also display unused sharding key generators with the <code>SHOW UNUSED SHARDING KEY GENERATORS</code>:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> SHOW SHARDING KEY GENERATORS;<br /> +--------------------------+-----------+-------+<br /> <span class="sy0">|</span> name                     <span class="sy0">|</span> <span class="kw3">type</span>      <span class="sy0">|</span> props <span class="sy0">|</span><br /> +--------------------------+-----------+-------+<br /> <span class="sy0">|</span> snowflake_key_generator  <span class="sy0">|</span> snowflake <span class="sy0">|</span> <span class="br0">{</span><span class="br0">}</span>    <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_detail_snowflake <span class="sy0">|</span> snowflake <span class="sy0">|</span> <span class="br0">{</span><span class="br0">}</span>    <span class="sy0">|</span><br /> +--------------------------+-----------+-------+<br /> <span class="nu0">2</span> rows <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.00</span> sec<span class="br0">)</span><br /> <br /> mysql<span class="sy0">&gt;</span> SHOW UNUSED SHARDING KEY GENERATORS;<br /> Empty <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.01</span> sec<span class="br0">)</span><br /> <br /> mysql<span class="sy0">&gt;</span> CREATE SHARDING KEY GENERATOR useless <span class="br0">(</span><br />     -<span class="sy0">&gt;</span> TYPE<span class="br0">(</span><span class="re2">NAME</span>=SNOWFLAKE<span class="br0">)</span><br />     -<span class="sy0">&gt;</span> <span class="br0">)</span>;<br /> Query OK, <span class="nu0">0</span> rows affected <span class="br0">(</span><span class="nu0">0.04</span> sec<span class="br0">)</span><br /> <br /> mysql<span class="sy0">&gt;</span> SHOW UNUSED SHARDING KEY GENERATORS;<br /> +---------+-----------+-------+<br /> <span class="sy0">|</span> name    <span class="sy0">|</span> <span class="kw3">type</span>      <span class="sy0">|</span> props <span class="sy0">|</span><br /> +---------+-----------+-------+<br /> <span class="sy0">|</span> useless <span class="sy0">|</span> snowflake <span class="sy0">|</span>       <span class="sy0">|</span><br /> +---------+-----------+-------+<br /> <span class="nu0">1</span> row <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.01</span> sec<span class="br0">)</span></div></div></pre><h3>Query unused sharding algorithm</h3> <p>DistSQL can reveal unused sharding algorithms with (you guessed it) the <code>SHOW UNUSED SHARDING ALGORITHMS</code> command:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> SHOW SHARDING ALGORITHMS;<br /> +--------------------------------+--------+-----------------------------------------------------+<br /> <span class="sy0">|</span> name                           <span class="sy0">|</span> <span class="kw3">type</span>   <span class="sy0">|</span> props                                               <span class="sy0">|</span><br /> +--------------------------------+--------+-----------------------------------------------------+<br /> <span class="sy0">|</span> database_inline                <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=ds_<span class="co1">${user_id % 2}</span>              <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_inline                 <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=t_order_<span class="co1">${order_id % 3}</span>        <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_item_inline            <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=t_order_item_<span class="co1">${order_id % 3}</span>   <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_detail_database_inline <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=ds_<span class="co1">${user_id % 2}</span>              <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_detail_table_inline    <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=t_order_detail_<span class="co1">${order_id % 3}</span> <span class="sy0">|</span><br /> +--------------------------------+--------+-----------------------------------------------------+<br /> <span class="nu0">5</span> rows <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.00</span> sec<span class="br0">)</span><br /> <br /> mysql<span class="sy0">&gt;</span> CREATE SHARDING ALGORITHM useless <span class="br0">(</span><br />     -<span class="sy0">&gt;</span> TYPE<span class="br0">(</span><span class="re2">NAME</span>=INLINE,PROPERTIES<span class="br0">(</span><span class="st0">"algorithm-expression"</span>=<span class="st0">"ds_<span class="es3">${user_id % 2}</span>"</span><span class="br0">)</span><span class="br0">)</span><br />     -<span class="sy0">&gt;</span> <span class="br0">)</span>;<br /> Query OK, <span class="nu0">0</span> rows affected <span class="br0">(</span><span class="nu0">0.04</span> sec<span class="br0">)</span><br /> <br /> mysql<span class="sy0">&gt;</span> SHOW UNUSED SHARDING ALGORITHMS;<br /> +---------+--------+----------------------------------------+<br /> <span class="sy0">|</span> name    <span class="sy0">|</span> <span class="kw3">type</span>   <span class="sy0">|</span> props                                  <span class="sy0">|</span><br /> +---------+--------+----------------------------------------+<br /> <span class="sy0">|</span> useless <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=ds_<span class="co1">${user_id % 2}</span> <span class="sy0">|</span><br /> +---------+--------+----------------------------------------+<br /> <span class="nu0">1</span> row <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.00</span> sec<span class="br0">)</span></div></div></pre><h3>Query rules that use the target storage resources</h3> <p>You can also see used resources within rules with <code>SHOW RULES USED RESOURCE</code>. All rules that use a resource can be queried, not limited to the sharding rule.</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> DROP RESOURCE ds_0;<br /> ERROR <span class="nu0">1101</span> <span class="br0">(</span>C1101<span class="br0">)</span>: Resource <span class="br0">[</span>ds_0<span class="br0">]</span> is still used by <span class="br0">[</span>ShardingRule<span class="br0">]</span>.<br /> <br /> mysql<span class="sy0">&gt;</span> SHOW RULES USED RESOURCE ds_0;<br /> +----------+----------------+<br /> <span class="sy0">|</span> <span class="kw3">type</span>     <span class="sy0">|</span> name           <span class="sy0">|</span><br /> +----------+----------------+<br /> <span class="sy0">|</span> sharding <span class="sy0">|</span> t_order        <span class="sy0">|</span><br /> <span class="sy0">|</span> sharding <span class="sy0">|</span> t_order_item   <span class="sy0">|</span><br /> <span class="sy0">|</span> sharding <span class="sy0">|</span> t_order_detail <span class="sy0">|</span><br /> +----------+----------------+<br /> <span class="nu0">3</span> rows <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.00</span> sec<span class="br0">)</span></div></div></pre><h3><strong>Query sharding rules that use the target primary key generator</strong></h3> <p>You can find sharding rules using a key generator with <code>SHOW SHARDING TABLE RULES USED KEY GENERATOR</code>:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> SHOW SHARDING KEY GENERATORS;<br /> +--------------------------+-----------+-------+<br /> <span class="sy0">|</span> name                     <span class="sy0">|</span> <span class="kw3">type</span>      <span class="sy0">|</span> props <span class="sy0">|</span><br /> +--------------------------+-----------+-------+<br /> <span class="sy0">|</span> snowflake_key_generator  <span class="sy0">|</span> snowflake <span class="sy0">|</span> <span class="br0">{</span><span class="br0">}</span>    <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_detail_snowflake <span class="sy0">|</span> snowflake <span class="sy0">|</span> <span class="br0">{</span><span class="br0">}</span>    <span class="sy0">|</span><br /> <span class="sy0">|</span> useless                  <span class="sy0">|</span> snowflake <span class="sy0">|</span> <span class="br0">{</span><span class="br0">}</span>    <span class="sy0">|</span><br /> +--------------------------+-----------+-------+<br /> <span class="nu0">3</span> rows <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.00</span> sec<span class="br0">)</span><br /> <br /> mysql<span class="sy0">&gt;</span> DROP SHARDING KEY GENERATOR snowflake_key_generator;<br /> ERROR <span class="nu0">1121</span> <span class="br0">(</span>C1121<span class="br0">)</span>: Sharding key generator <span class="sy0">`</span><span class="br0">[</span>snowflake_key_generator<span class="br0">]</span><span class="sy0">`</span> <span class="kw1">in</span> database <span class="sy0">`</span>sharding_db<span class="sy0">`</span> are still <span class="kw1">in</span> used.<br /> <br /> mysql<span class="sy0">&gt;</span> SHOW SHARDING TABLE RULES USED KEY GENERATOR snowflake_key_generator;<br /> +-------+--------------+<br /> <span class="sy0">|</span> <span class="kw3">type</span>  <span class="sy0">|</span> name         <span class="sy0">|</span><br /> +-------+--------------+<br /> <span class="sy0">|</span> table <span class="sy0">|</span> t_order      <span class="sy0">|</span><br /> <span class="sy0">|</span> table <span class="sy0">|</span> t_order_item <span class="sy0">|</span><br /> +-------+--------------+<br /> <span class="nu0">2</span> rows <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.00</span> sec<span class="br0">)</span></div></div></pre><h3>Query sharding rules that use the target algorithm</h3> <p>Show sharding rules using a target algorithm with <code>SHOW SHARDING TABLE RULES USED ALGORITHM</code>:</p> <pre> <div class="geshifilter"><div class="bash geshifilter-bash">mysql<span class="sy0">&gt;</span> SHOW SHARDING ALGORITHMS;<br /> +--------------------------------+--------+-----------------------------------------------------+<br /> <span class="sy0">|</span> name                           <span class="sy0">|</span> <span class="kw3">type</span>   <span class="sy0">|</span> props                                               <span class="sy0">|</span><br /> +--------------------------------+--------+-----------------------------------------------------+<br /> <span class="sy0">|</span> database_inline                <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=ds_<span class="co1">${user_id % 2}</span>              <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_inline                 <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=t_order_<span class="co1">${order_id % 3}</span>        <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_item_inline            <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=t_order_item_<span class="co1">${order_id % 3}</span>   <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_detail_database_inline <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=ds_<span class="co1">${user_id % 2}</span>              <span class="sy0">|</span><br /> <span class="sy0">|</span> t_order_detail_table_inline    <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=t_order_detail_<span class="co1">${order_id % 3}</span> <span class="sy0">|</span><br /> <span class="sy0">|</span> useless                        <span class="sy0">|</span> inline <span class="sy0">|</span> algorithm-expression=ds_<span class="co1">${user_id % 2}</span>              <span class="sy0">|</span><br /> +--------------------------------+--------+-----------------------------------------------------+<br /> <span class="nu0">6</span> rows <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.00</span> sec<span class="br0">)</span><br /> <br /> mysql<span class="sy0">&gt;</span> DROP SHARDING ALGORITHM t_order_detail_table_inline;<br /> ERROR <span class="nu0">1116</span> <span class="br0">(</span>C1116<span class="br0">)</span>: Sharding algorithms <span class="sy0">`</span><span class="br0">[</span>t_order_detail_table_inline<span class="br0">]</span><span class="sy0">`</span> <span class="kw1">in</span> database <span class="sy0">`</span>sharding_db<span class="sy0">`</span> are still <span class="kw1">in</span> used.<br /> <br /> mysql<span class="sy0">&gt;</span> SHOW SHARDING TABLE RULES USED ALGORITHM t_order_detail_table_inline;<br /> +-------+----------------+<br /> <span class="sy0">|</span> <span class="kw3">type</span>  <span class="sy0">|</span> name           <span class="sy0">|</span><br /> +-------+----------------+<br /> <span class="sy0">|</span> table <span class="sy0">|</span> t_order_detail <span class="sy0">|</span><br /> +-------+----------------+<br /> <span class="nu0">1</span> row <span class="kw1">in</span> <span class="kw1">set</span> <span class="br0">(</span><span class="nu0">0.00</span> sec<span class="br0">)</span></div></div></pre><h2>Make sharding better</h2> <p>DistSQL provides a flexible syntax to help simplify operations. In addition to the <strong>INLINE</strong> algorithm, DistSQL supports standard sharding, compound sharding, HINT sharding, and custom sharding algorithms.</p> <p>If you have any questions or suggestions about <a href="https://shardingsphere.apache.org/" target="_blank">Apache ShardingSphere</a>, please feel free to post them on <a href="https://github.com/apache/shardingsphere" target="_blank">ShardingSphereGitHub</a><u>.</u></p> </div> <div class="clearfix text-formatted field field--name-field-article-subhead field--type-text-long field--label-hidden field__item"><p>Take a look at a data sharding scenario in which DistSQL&#039;s flexibility allows you to create a distributed database.</p> </div> <div class="field field--name-field-lead-image field--type-entity-reference field--label-hidden field__item"><article class="media media--type-image media--view-mode-caption"> <div class="field field--name-field-media-image field--type-image field--label-hidden field__item"> <img loading="lazy" src="/sites/default/files/lead-images/BIZ_darwincloud_520x292_0311LL.png" width="520" height="292" alt="diagram of planning a cloud" title="diagram of planning a cloud" /> </div> <div class="field field--name-field-caption field--type-text-long field--label-hidden caption field__item"><span class="caption__byline">Image by: </span><p>Opensource.com</p> </div> </article> </div> <div class="field field--name-field-tags field--type-entity-reference field--label-hidden field__items"> <div class="field__item"><a href="/tags/databases" hreflang="en">Databases</a></div> </div> <div class="hidden field field--name-field-listicle-title field--type-string field--label-hidden field__item">What to read next</div> <div class="field field--name-field-default-license field--type-list-string field--label-hidden field__item"><a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"> <img alt="Creative Commons License" src="/themes/osdc/assets/img/cc-by-sa-4.png" title="This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License." /></a>This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.</div> <section id='comments' class="field field--name-field-comments field--type-comment field--label-hidden comment-wrapper"> <div class="comments__count"> <div class="login"><a href="/user/register?absolute=1">Register</a> or <a href="/user/login?current=/rss.xml&amp;absolute=1">Login</a> to post a comment.</div> </div> </section> Fri, 23 Sep 2022 07:00:00 +0000 Raigor Jiang 70205 at https://opensource.com https://opensource.com/article/22/9/dynamic-distributed-database-distsql#comments