<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>Shesh&#39;s blog</title>
  
  
  <link href="/atom.xml" rel="self"/>
  
  <link href="https://www.sheshbabu.com/"/>
  <updated>2024-08-18T13:38:09.087Z</updated>
  <id>https://www.sheshbabu.com/</id>
  
  <author>
    <name>Sheshbabu Chinnakonda</name>
    
  </author>
  
  <generator uri="http://hexo.io/">Hexo</generator>
  
  <entry>
    <title>The Case of the Crashing Checkout Flows: My Strangest Debugging Story</title>
    <link href="https://www.sheshbabu.com/posts/my-binary-search-debugging-story/"/>
    <id>https://www.sheshbabu.com/posts/my-binary-search-debugging-story/</id>
    <published>2024-08-18T07:54:34.000Z</published>
    <updated>2024-08-18T13:38:09.087Z</updated>
    
    <content type="html"><![CDATA[<p>The year was 2015, and I had just joined RedMart.</p><p>I was very impressed by the importance RedMart placed on addressing customer feedback. We had a customer feedback form, and the responses were sent to everyone’s email inbox. We would triage the feedback, and if there were any software related issues, my team would pick them up.</p><p>At the end of that year, we started seeing a strange pattern in these feedback messages. Some users complained they couldn’t complete checkout, while others reported their browsers crashing during the process.</p><p><em>Note: If you’re not familiar with the term “checkout flow”, it’s the series of pages you need to go through when you’re ordering from an ecommerce shop. Like filling up your address, choosing your payment method etc.</em></p><h2 id="Memory-Leaks"><a href="#Memory-Leaks" class="headerlink" title="Memory Leaks?"></a>Memory Leaks?</h2><p>I started investigating this issue. One thing the feedback had in common was that they were all from MacBooks via Chrome browser (detected using useragent string). The <a href="https://support.apple.com/en-us/111956" target="_blank" rel="noopener">base model MacBook Air</a> at that time had 4GB of memory. I started suspecting it might be a memory leak.</p><p>In 2015, RedMart’s desktop webapp (aka “Golden Grocer” or “GG” internally) was written in Backbone.js and jQuery. We had started incrementally migrating the pages to React, but the checkout flow was still in Backbone and jQuery.</p><p>I had some past experience debugging memory leaks in JVM, so I fired up the Chrome Devtools and <a href="https://developer.chrome.com/docs/devtools/memory-problems" target="_blank" rel="noopener">started profiling</a>. I went through the checkout flow and found that there were about ~16MB of objects leaking memory. These were likely JS closures or Backbone/jQuery event handlers not cleaned up properly. But 16MB shouldn’t be enough to crash an entire browser. </p><p>In software debugging, there’s a saying:</p><blockquote><p>If you can reproduce a bug consistently, you’ve already solved half the problem.</p></blockquote><p>I’m not sure where I first heard this, but if you’re not able to reproduce a bug, then you can’t fix it properly. In this situation, I know it happened during checkout flow but I wasn’t able to get the browser to crash. For a couple of days, I use the Chrome Devtools to go through the checkout flow again and again trying to see if I could get it to produce a big memory leak.</p><p>This didn’t work out, so I wanted to try a different approach. Instead of using the fine-grained Chrome Devtools memory profiler, I opened the Chrome Task Manager. I noted down GG’s memory consumption on the home page - it was around 300MB. I went through the checkout flow. The first page was address selection, the memory didn’t budge much. The second was slots selection, memory remained the same. The third was payment selection, and the memory jumped to more than 1.5GB. </p><p><em>Bingo!</em></p><p>Something in payment page was causing the memory to jump from 300MB to more than 1.5GB. But there’s nothing special there in terms of code.</p><h2 id="Binary-Search-Debugging"><a href="#Binary-Search-Debugging" class="headerlink" title="Binary Search Debugging"></a>Binary Search Debugging</h2><p>Since there’s nothing wrong with the code, we can’t fix this issue using normal debugging techniques. We need to use a different approach. I’ve used this approach called “Binary Search Debugging” in the past and it’s very effective at locating the bugs. </p><p>Here’s how it works: We first divide our codebase into two parts. We comment out one part and check if we can reproduce the bug. If we can’t reproduce the bug, we then comment out the other part while uncommenting the first part. We then check if we can reproduce the bug. The bug is in the part that remains uncommented. Once we identify the problematic part, we further divide it into two and repeat the process. Eventually, we will pinpoint the specific line of code causing the bug.</p><p>This is very effective method and I’ve successfully used it many times in the past. But this time, I was not able to find the code that caused this bug. </p><p>I have been using this approach on the JS code, but I wondered if it’s a CSS issue. It sounds ridiculous, but the whole situation is ridiculous.</p><p>I did the same for CSS and I was actually able to find the bug! When a payment method is selected, we show a pulsing blue glow effect around that option. This was done using <code>box-shadow</code> and <code>animation</code>, and once I comment out the animation, the memory increase didn’t happen. I searched the web for related issues but I was not able to find it.</p><p>Finally, I was able to fix the memory issue that blocked checkout by just disabling a CSS glow animation. It wasn’t a code issue, it was a browser issue!</p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>It was an interesting experience that required a combination of both skill and luck to fix the bug. I had the memory profiling skill and knowledge about the binary search debugging technique. However, it was also random luck that made me open the Chrome Task Manager and apply the debugging technique to CSS files.</p><p>Thanks for reading! :)</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;The year was 2015, and I had just joined RedMart.&lt;/p&gt;
&lt;p&gt;I was very impressed by the importance RedMart placed on addressing customer fee
      
    
    </summary>
    
    
      <category term="Development" scheme="https://www.sheshbabu.com/tags/Development/"/>
    
      <category term="Testing" scheme="https://www.sheshbabu.com/tags/Testing/"/>
    
  </entry>
  
  <entry>
    <title>Swiss Cheese Model of Software Reliability</title>
    <link href="https://www.sheshbabu.com/posts/swiss-cheese-model-of-software-reliability/"/>
    <id>https://www.sheshbabu.com/posts/swiss-cheese-model-of-software-reliability/</id>
    <published>2024-08-17T07:47:18.000Z</published>
    <updated>2024-08-18T07:11:30.511Z</updated>
    
    <content type="html"><![CDATA[<p>Software surrounds us. It’s in the planes we fly and the cars we drive. It handles our finances, helps us learn, and keeps us connected. Despite being an integral part of our everyday lives, we still encounter software failures with regular frequency. Why is this? How do we eliminate software bugs?</p><p>In any complex system like the weather, ecology, or even the human body, there are countless variables, triggers, and participants that are interconnected. It’s hard to accurately predict the future state of such a system. Most software, while not as complex as weather patterns or ecosystems, still presents similar challenges. There are so many ways code can behave that the more complex the software system becomes, the harder it is to ensure it’s 100% bug-free.</p><p>How do we deal with this then? Should we just give up, saying that 100% bug-free software is not possible? Of course not! We try our best and use different approaches to reduce the number of bugs. But which approaches should we use?</p><p><img src="/images/2024-swiss-cheese-model-of-software-reliability/2024-swiss-cheese-model-of-software-reliability.png" alt=""></p><h2 id="Silver-Bullet-Model"><a href="#Silver-Bullet-Model" class="headerlink" title="Silver Bullet Model"></a>Silver Bullet Model</h2><p>Here’s where it gets confusing. Every developer, team, or devtools vendor would propose a different list of approaches. If you’ve been in this profession long enough, you might have encountered the heated debates between statically-typed and dynamically-typed languages, functional vs object-oriented programming models, and so on. </p><p>As engineers, we think of ourselves as objective, but most of us wear the tools we use as our identities. If you work with statically-typed languages like Java, Go, and C, you criticize developers using JavaScript or Python, saying that using dynamically-typed languages is bad for reliability. Similarly, advocates of TDD say that if you’re not following the TDD approach, your software is not good. </p><p>Does that mean that software written in statically-typed languages or following TDD result in no bugs? Far from it. </p><h2 id="Swiss-Cheese-Model"><a href="#Swiss-Cheese-Model" class="headerlink" title="Swiss Cheese Model"></a>Swiss Cheese Model</h2><p>Fred Brooks, the author of <em>The Mythical Man-Month</em>, wrote a paper titled <a href="https://www.cs.unc.edu/techreports/86-020.pdf" target="_blank" rel="noopener">No Silver Bullet — Essence and Accident in Software Engineering</a>, in which he says:</p><blockquote><p>There is no single development, in either technology or management technique, which by itself promises even one order of magnitude improvement in productivity, in reliability, in simplicity.</p></blockquote><p>This was written almost four decades back, and it’s still true! So, if there’s no silver bullet, how do we make software more reliable? </p><p>What we can do instead is to use a layered strategy, instead of relying on a single approach to increase software reliability.</p><p>This strategy stacks multiple approaches one after another. As the software passes through these layers, some bugs are caught in the initial layers, while the remaining bugs are trapped in subsequent layers. By the time the software reaches the end user, the number of bugs is significantly reduced.</p><p>This layered approach is commonly known as <a href="https://en.wikipedia.org/wiki/Defense_in_depth_(computing)" target="_blank" rel="noopener">Defense in Depth</a> in cybersecurity or as <a href="https://en.wikipedia.org/wiki/Swiss_cheese_model" target="_blank" rel="noopener">Swiss Cheese Model</a> in disease prevention.</p><p>I particularly like the <a href="https://www.google.com/search?q=swiss+cheese+slice" target="_blank" rel="noopener">Swiss cheese imagery</a> because it’s not only fun, but the holes in the cheese represent the ways in which these individual layers fail. </p><p>Approaches like static typing, unit tests, linting, code reviews, canary deployment, or code coverage, are not 100% effective at catching bugs on their own. However, if we stack them together, bugs would be caught in one of these layers and would greatly improve the reliability of software that users use. </p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>As we can see, relying on a single approach to prevent bugs is not a good idea. Adopting a layered strategy ensures that bugs are caught in any one of these layers and it also makes the process future-proof. If there are new innovations in future, like <em>AI Code Review</em> tools, we can add them as one of the layers, and thereby increasing the reliability even more.</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;Software surrounds us. It’s in the planes we fly and the cars we drive. It handles our finances, helps us learn, and keeps us connected. 
      
    
    </summary>
    
    
      <category term="Development" scheme="https://www.sheshbabu.com/tags/Development/"/>
    
      <category term="Testing" scheme="https://www.sheshbabu.com/tags/Testing/"/>
    
      <category term="Code Reviews" scheme="https://www.sheshbabu.com/tags/Code-Reviews/"/>
    
      <category term="Maintainability" scheme="https://www.sheshbabu.com/tags/Maintainability/"/>
    
      <category term="Reliability" scheme="https://www.sheshbabu.com/tags/Reliability/"/>
    
      <category term="Mental Model" scheme="https://www.sheshbabu.com/tags/Mental-Model/"/>
    
  </entry>
  
  <entry>
    <title>Zero-Downtime Postgres Credentials Rotation with Node.js</title>
    <link href="https://www.sheshbabu.com/posts/implementing-zero-downtime-postgres-credentials-rotation-with-node-js/"/>
    <id>https://www.sheshbabu.com/posts/implementing-zero-downtime-postgres-credentials-rotation-with-node-js/</id>
    <published>2024-08-13T14:03:15.000Z</published>
    <updated>2024-08-14T14:15:49.758Z</updated>
    
    <content type="html"><![CDATA[<p>In enterprise environments, it’s common to store database credentials in a vault like AWS Secrets Manager. These credentials are usually rotated every few weeks based on the company’s security policy. </p><p>While this improves security, it adds complexity to the application. Instead of pulling these credentials from environment variables, the application code must now query AWS Secrets Manager and handle cases where database connections are dropped because of credential rotation.</p><p>How does one update the database credentials from vault if they are rotated while the application is running?</p><p><strong>Manual Approach:</strong> We can note the date and time when the next credential rotation is scheduled. Before the rotation, we can take the application offline and then restart it after the rotation has completed. This approach is simple but involves downtime. If you’re in an enterprise environment, you likely have many databases and applications, and this approach might not be feasible.</p><p><strong>Brute-force Approach:</strong> Before each database query, retrieve the latest set of credentials from vault and then use that credentials to establish the connection to database. This approach works, but it’s very wasteful and would degrade the performance. It will also cost a lot since AWS <a href="https://aws.amazon.com/secrets-manager/pricing/" target="_blank" rel="noopener">charges for API calls</a> to Secrets Manager.</p><h2 id="Better-Approach"><a href="#Better-Approach" class="headerlink" title="Better Approach"></a>Better Approach</h2><p>Is there a better approach? Instead of querying the vault for credentials before each database connection, wouldn’t it be more efficient if we fetch the credentials from vault during application startup, store them in memory, and then refresh them after they get rotated? This would be ideal, but how would the application know that the credentials have rotated?</p><p>The key to solving this is to understand what happens to connections when credentials get rotated. Once a connection is established using a set of credentials, it will work until it gets terminated. So, if a connection is used to execute a query, and while it’s under execution, the credentials got rotated, the connection will still work! It won’t get cut-off in the middle. </p><p>In complex applications, connections are managed in a pool. If the pool is empty and a query needs to be run, then a new connection is established and used to run the query. After running this query, the connection is released back into the pool. It’s not terminated until the idle timeout has passed. So, the connections established using the old credentials can still be used if they’re present in the pool.</p><p>However, establishing new connections with old credentials would fail. We can catch these connection errors, then fetch new credentials from the vault and reestablish the connection.</p><h2 id="Implementation"><a href="#Implementation" class="headerlink" title="Implementation"></a>Implementation</h2><p>Let’s start with a simple implementation using <a href="https://node-postgres.com" target="_blank" rel="noopener">node-postgres</a>:</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// ./database.js</span><span class="token keyword">import</span> pg <span class="token keyword">from</span> <span class="token string">"pg"</span><span class="token punctuation">;</span><span class="token keyword">const</span> OLD_CONNECTION_STRING <span class="token operator">=</span> <span class="token string">"postgres://postgres:postgres@localhost:5432/postgres?sslmode=disable"</span><span class="token punctuation">;</span><span class="token keyword">const</span> NEW_CONNECTION_STRING <span class="token operator">=</span> <span class="token string">"postgres://postgres:new_password@localhost:5432/postgres?sslmode=disable"</span><span class="token punctuation">;</span><span class="token keyword">let</span> pool <span class="token operator">=</span> <span class="token keyword">null</span><span class="token punctuation">;</span><span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">query</span><span class="token punctuation">(</span>query<span class="token punctuation">,</span> params<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">if</span> <span class="token punctuation">(</span>pool <span class="token operator">===</span> <span class="token keyword">null</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"creating new pool"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    pool <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">pg<span class="token punctuation">.</span>Pool</span><span class="token punctuation">(</span><span class="token punctuation">{</span> connectionString<span class="token punctuation">:</span> OLD_CONNECTION_STRING <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span>  <span class="token keyword">const</span> client <span class="token operator">=</span> <span class="token keyword">await</span> pool<span class="token punctuation">.</span><span class="token function">connect</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  client <span class="token operator">=</span> <span class="token keyword">await</span> pool<span class="token punctuation">.</span><span class="token function">connect</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">try</span> <span class="token punctuation">{</span>    <span class="token keyword">return</span> <span class="token keyword">await</span> client<span class="token punctuation">.</span><span class="token function">query</span><span class="token punctuation">(</span>query<span class="token punctuation">,</span> params<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span> <span class="token keyword">finally</span> <span class="token punctuation">{</span>    client<span class="token punctuation">.</span><span class="token function">release</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token keyword">export</span> <span class="token keyword">default</span> <span class="token punctuation">{</span> query <span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre><p>This can be used as follows:</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// ./index.js</span><span class="token keyword">import</span> Database <span class="token keyword">from</span> <span class="token string">"./database.js"</span><span class="token punctuation">;</span><span class="token keyword">await</span> Database<span class="token punctuation">.</span><span class="token function">query</span><span class="token punctuation">(</span><span class="token template-string"><span class="token string">`    CREATE TABLE IF NOT EXISTS users (        id SERIAL PRIMARY KEY,        name TEXT NOT NULL,        email TEXT NOT NULL    )`</span></span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">let</span> i <span class="token operator">=</span> <span class="token number">0</span><span class="token punctuation">;</span> i <span class="token operator">&lt;</span> <span class="token number">100</span><span class="token punctuation">;</span> i<span class="token operator">++</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">await</span> Database<span class="token punctuation">.</span><span class="token function">query</span><span class="token punctuation">(</span><span class="token string">"INSERT INTO users (name, email) VALUES ($1, $2)"</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token template-string"><span class="token string">`user_</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>i<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">`</span></span><span class="token punctuation">,</span> <span class="token template-string"><span class="token string">`user_</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>i<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">@email.com`</span></span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">await</span> <span class="token function">sleep</span><span class="token punctuation">(</span><span class="token number">1000</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">function</span> <span class="token function">sleep</span><span class="token punctuation">(</span>ms<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">return</span> <span class="token keyword">new</span> <span class="token class-name">Promise</span><span class="token punctuation">(</span><span class="token punctuation">(</span>resolve<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token function">setTimeout</span><span class="token punctuation">(</span>resolve<span class="token punctuation">,</span> ms<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>This creates a table and inserts 100 records in a loop. I’ve added a <code>sleep</code> so the loop takes 1-2 minutes to run.</p><p>Let’s run this, and while it’s running, we can change the database credentials by running this query:</p><pre class=" language-sql"><code class="language-sql"><span class="token keyword">ALTER</span> <span class="token keyword">USER</span> postgres <span class="token keyword">WITH</span> PASSWORD <span class="token string">'new_password'</span><span class="token punctuation">;</span></code></pre><p>The script errors out and exits. If we look at the stack trace, we can see the error object has an attribute called <code>routine</code> with value <code>auth_failed</code>.</p><p>We can catch this error while connecting, fetch new credentials from vault and then re-create the pool with this new credentials:</p><pre class=" language-diff"><code class="language-diff">  // ./database.js  import pg from "pg";  const OLD_CONNECTION_STRING = "postgres://postgres:postgres@localhost:5432/postgres?sslmode=disable";  const NEW_CONNECTION_STRING = "postgres://postgres:new_password@localhost:5432/postgres?sslmode=disable";  let pool = null;  async function query(query, params) {    if (pool === null) {      console.log("creating new pool");      pool = new pg.Pool({ connectionString: OLD_CONNECTION_STRING });    }<span class="token deleted">-   const client = await pool.connect();</span><span class="token deleted">-   client = await pool.connect();</span><span class="token inserted">+   let client;</span><span class="token inserted">+   try {</span><span class="token inserted">+     client = await pool.connect();</span><span class="token inserted">+   } catch (e) {</span><span class="token inserted">+     if (e.routine !== "auth_failed") {</span><span class="token inserted">+       throw e;</span><span class="token inserted">+     }</span><span class="token inserted">+     console.log("auth failed, trying new password")</span><span class="token inserted">+     pool = new pg.Pool({ connectionString: NEW_CONNECTION_STRING });</span><span class="token inserted">+     client = await pool.connect();</span><span class="token inserted">+   }</span>    try {      return await client.query(query, params);    } finally {      client.release();    }  }  export default { query };</code></pre><p>Now, if you reset the password to the old one, and then try the same flow again, you’ll quickly see a <code>&quot;auth failed, trying new password&quot;</code> log and the script carries on without any interruptions.</p><p>I’ve used the <code>NEW_CONNECTION_STRING</code> from a variable, but you can update the above code so it fetches this from AWS Secrets Manager, Azure Key Vault, etc. It’s also advisable to throttle the calls to these vault APIs so if there are multiple connections that are erroring out, there’s only one call that retrieves the new credentials and creates a new pool.</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;In enterprise environments, it’s common to store database credentials in a vault like AWS Secrets Manager. These credentials are usually 
      
    
    </summary>
    
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
      <category term="Node.js" scheme="https://www.sheshbabu.com/tags/Node-js/"/>
    
      <category term="Postgres" scheme="https://www.sheshbabu.com/tags/Postgres/"/>
    
  </entry>
  
  <entry>
    <title>Implementing SAML SSO in Node.js with Microsoft Entra ID</title>
    <link href="https://www.sheshbabu.com/posts/implementing-saml-authentication-in-node-js/"/>
    <id>https://www.sheshbabu.com/posts/implementing-saml-authentication-in-node-js/</id>
    <published>2024-08-11T14:11:30.000Z</published>
    <updated>2024-08-12T10:42:44.656Z</updated>
    
    <content type="html"><![CDATA[<p>SAML is one of the commonly used standards for implementing SSO in enterprise environments. Even though OIDC is rapidly gaining traction, not everyone supports it, or there are compliance requirements that mandate SAML.</p><p><img src="/images/2020-saml/image-2.png" alt=""></p><p>For a conceptual overview of how the SAML flow works, please refer to <a href="https://www.sheshbabu.com/posts/visual-explanation-of-saml-authentication/">this post</a>.</p><p>We’ll use <a href="https://learn.microsoft.com/en-sg/entra/fundamentals/new-name?culture=en-sg&country=sg" target="_blank" rel="noopener">Microsoft Entra ID</a> (Azure Active Directory) as the Identity Provider (IdP).</p><p>Let’s start with a simple Express example:</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// index.js</span><span class="token keyword">import</span> express <span class="token keyword">from</span> <span class="token string">"express"</span><span class="token punctuation">;</span><span class="token keyword">const</span> app <span class="token operator">=</span> <span class="token function">express</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token keyword">const</span> router <span class="token operator">=</span> express<span class="token punctuation">.</span><span class="token function">Router</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>router<span class="token punctuation">.</span><span class="token keyword">get</span><span class="token punctuation">(</span><span class="token string">"/public"</span><span class="token punctuation">,</span> <span class="token punctuation">(</span>req<span class="token punctuation">,</span> res<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span>  res<span class="token punctuation">.</span><span class="token function">send</span><span class="token punctuation">(</span><span class="token string">"Please login"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>router<span class="token punctuation">.</span><span class="token keyword">get</span><span class="token punctuation">(</span><span class="token string">"/private"</span><span class="token punctuation">,</span> <span class="token punctuation">(</span>req<span class="token punctuation">,</span> res<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span>  res<span class="token punctuation">.</span><span class="token function">send</span><span class="token punctuation">(</span><span class="token string">"Welcome home!"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>app<span class="token punctuation">.</span><span class="token function">use</span><span class="token punctuation">(</span><span class="token string">"/api"</span><span class="token punctuation">,</span> router<span class="token punctuation">)</span><span class="token punctuation">;</span>app<span class="token punctuation">.</span><span class="token function">listen</span><span class="token punctuation">(</span><span class="token number">3000</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span>  console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"Server is running on port 3000"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>We’ve two routes: <code>/api/public</code> is public and can be seen by anyone, <code>/api/private</code> is private and should only be visible to logged-in users. If the IdP requires the server to use HTTPS, please refer to <a href="https://www.sheshbabu.com/posts/running-express-over-https-in-localhost/">this post</a> on how to implement HTTPS in Express.</p><p>We’ll add the SAML auth logic in a separate middleware module to keep things clean. Let’s start with a dummy implementation:</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// saml.js</span><span class="token keyword">import</span> express <span class="token keyword">from</span> <span class="token string">"express"</span><span class="token punctuation">;</span><span class="token keyword">const</span> router <span class="token operator">=</span> express<span class="token punctuation">.</span><span class="token function">Router</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>router<span class="token punctuation">.</span><span class="token keyword">get</span><span class="token punctuation">(</span><span class="token string">"/login"</span><span class="token punctuation">,</span> <span class="token punctuation">(</span>req<span class="token punctuation">,</span> res<span class="token punctuation">,</span> next<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span>  res<span class="token punctuation">.</span><span class="token function">redirect</span><span class="token punctuation">(</span><span class="token string">"/api/private"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>router<span class="token punctuation">.</span><span class="token function">post</span><span class="token punctuation">(</span><span class="token string">"/login/callback"</span><span class="token punctuation">,</span> <span class="token punctuation">(</span>req<span class="token punctuation">,</span> res<span class="token punctuation">,</span> next<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span>  res<span class="token punctuation">.</span><span class="token function">redirect</span><span class="token punctuation">(</span><span class="token string">"/api/private"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token keyword">export</span> <span class="token keyword">default</span> router<span class="token punctuation">;</span></code></pre><p>Update <code>index.js</code> to use the SAML middleware.</p><pre class=" language-diff"><code class="language-diff">  // index.js  import express from "express";<span class="token inserted">+ import SamlAuth from "./saml.js";</span>  const app = express();<span class="token inserted">+ app.use(SamlAuth);</span>  const router = express.Router();  router.get("/public", (req, res) => {    res.send("Please login");  });<span class="token inserted">+ router.use(ensureLoggedIn);</span>  router.get("/private", (req, res) => {    res.send("Welcome home!");  });  app.use("/api", router);<span class="token inserted">+ function ensureLoggedIn(req, res, next) {</span><span class="token inserted">+   if (!req.isAuthenticated()) {</span><span class="token inserted">+     return res.redirect("/login");</span><span class="token inserted">+   } else {</span><span class="token inserted">+     next();</span><span class="token inserted">+   }</span><span class="token inserted">+ }</span>  app.listen(3000, () => {    console.log("Server is running on port 3000");  });</code></pre><p>We’ve also added an <code>ensureLoggedIn</code> middleware to validate whether an user is logged-in for the private routes and redirect them to login page if they’re not logged-in.</p><p>Next, we will configure <code>passport</code> and <code>passport-saml</code> in <code>saml.js</code>:</p><pre class=" language-diff"><code class="language-diff">  // saml.js  import express from "express";<span class="token inserted">+ import passport from "passport";</span><span class="token inserted">+ import { Strategy } from "passport-saml";</span><span class="token inserted">+ const SAML_ISSUER = "xxx";</span><span class="token inserted">+ const SAML_CERT = "xxx";</span><span class="token inserted">+ const config = {</span><span class="token inserted">+   entryPoint: "https://login.microsoftonline.com/{tenantId}/saml2", // replace 'tenant' before using</span><span class="token inserted">+   callbackUrl: "http://localhost:3000/login/callback",</span><span class="token inserted">+   issuer: SAML_ISSUER,</span><span class="token inserted">+   cert: SAML_CERT,</span><span class="token inserted">+   logoutUrl: "/logout"</span><span class="token inserted">+ };</span>  const router = express.Router();<span class="token inserted">+ passport.use(</span><span class="token inserted">+   new Strategy(config, (profile, cb) => {</span><span class="token inserted">+     const email = profile["http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name"];</span><span class="token inserted">+     const name = profile["http://schemas.microsoft.com/identity/claims/displayname"];</span><span class="token inserted">+     return cb(null, { email, name });</span><span class="token inserted">+   })</span><span class="token inserted">+ );</span>  router.get("/login", (req, res, next) => {    res.redirect("/api/private");  });  router.post("/login/callback", (req, res, next) => {    res.redirect("/api/private");  });  export default router;</code></pre><p>The values for <code>tenantId</code>, <code>SAML_ISSUER</code>, and <code>SAML_CERT</code> would be available while registering your application in Microsoft Entra ID.</p><p>We’ll then <a href="https://www.passportjs.org/tutorials/password/session/" target="_blank" rel="noopener">configure the session</a> so the users don’t need to keep re-logging-in. I’ve used SQLite as session store, but you can replace it with Redis, Postgres etc.</p><pre class=" language-diff"><code class="language-diff">  // saml.js  import express from "express";  import passport from "passport";  import { Strategy } from "passport-saml";<span class="token inserted">+ import session from "express-session";</span><span class="token inserted">+ import connectSqlite3 from "connect-sqlite3";</span><span class="token inserted">+ import cookieParser from "cookie-parser";</span><span class="token inserted">+ const SQLiteStore = connectSqlite3(session);</span><span class="token inserted">+ const SESSION_SECRET = "xxx";</span>  const SAML_ISSUER = "xxx";  const SAML_CERT = "xxx";  const config = {    entryPoint: "https://login.microsoftonline.com/{tenantId}/saml2", // replace 'tenant' before using    callbackUrl: "http://localhost:3000/login/callback",    issuer: SAML_ISSUER,    cert: SAML_CERT,    logoutUrl: "/logout"  };  const router = express.Router();  passport.use(    new Strategy(config, (profile, cb) => {      const email = profile["http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name"];      const name = profile["http://schemas.microsoft.com/identity/claims/displayname"];      return cb(null, { email, name });    })  );<span class="token inserted">+ passport.serializeUser((user, cb) => {</span><span class="token inserted">+   cb(null, user);</span><span class="token inserted">+ });</span><span class="token inserted">+ passport.deserializeUser((user, cb) => {</span><span class="token inserted">+   cb(null, user);</span><span class="token inserted">+ });</span><span class="token inserted">+ router.use(cookieParser());</span><span class="token inserted">+ router.use(</span><span class="token inserted">+   session({</span><span class="token inserted">+     secret: SESSION_SECRET,</span><span class="token inserted">+     resave: false,</span><span class="token inserted">+     saveUninitialized: false,</span><span class="token inserted">+     store: new SQLiteStore({ db: "sessions.db" })</span><span class="token inserted">+   })</span><span class="token inserted">+ );</span><span class="token inserted">+ router.use(passport.initialize());</span><span class="token inserted">+ router.use(passport.session());</span>  router.get("/login", (req, res, next) => {    res.redirect("/api/private");  });  router.post("/login/callback", (req, res, next) => {    res.redirect("/api/private");  });  export default router;</code></pre><p>We’ll update the <code>/login/*</code> endpoints and also add a logout endpoint</p><pre class=" language-diff"><code class="language-diff">  // saml.js  import express from "express";  import passport from "passport";  import { Strategy } from "passport-saml";  import session from "express-session";  import connectSqlite3 from "connect-sqlite3";  import cookieParser from "cookie-parser";  const SQLiteStore = connectSqlite3(session);  const SESSION_SECRET = "xxx";  const SAML_ISSUER = "xxx";  const SAML_CERT = "xxx";  const config = {    entryPoint: "https://login.microsoftonline.com/{tenantId}/saml2", // replace 'tenant' before using    callbackUrl: "http://localhost:3000/login/callback",    issuer: SAML_ISSUER,    cert: SAML_CERT,    logoutUrl: "/logout"  };  const router = express.Router();  passport.use(    new Strategy(config, (profile, cb) => {      const email = profile["http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name"];      const name = profile["http://schemas.microsoft.com/identity/claims/displayname"];      return cb(null, { email, name });    })  );  passport.serializeUser((user, cb) => {    cb(null, user);  });  passport.deserializeUser((user, cb) => {    cb(null, user);  });  router.use(cookieParser());  router.use(    session({      secret: SESSION_SECRET,      resave: false,      saveUninitialized: false,      store: new SQLiteStore({ db: "sessions.db" }),    })  );  router.use(passport.initialize());  router.use(passport.session());<span class="token deleted">- router.get("/login", (req, res, next) => {</span><span class="token deleted">-   res.redirect("/api/private");</span><span class="token deleted">- });</span><span class="token inserted">+ router.get("/login", passport.authenticate("saml"));</span><span class="token deleted">- router.post("/login/callback", (req, res, next) => {</span><span class="token deleted">-   res.redirect("/api/private");</span><span class="token deleted">- });</span><span class="token inserted">+ router.post("/login/callback", express.urlencoded({ extended: false }), passport.authenticate("saml"), (req, res) => {</span><span class="token inserted">+   res.redirect("/api/private");</span><span class="token inserted">+ });</span><span class="token inserted">+ router.post("/logout", (req, res, next) => {</span><span class="token inserted">+   req.logout((err) => {</span><span class="token inserted">+     if (err) {</span><span class="token inserted">+       return next(err);</span><span class="token inserted">+     }</span><span class="token inserted">+     req.session.destroy(() => {</span><span class="token inserted">+       res.clearCookie("connect.sid");</span><span class="token inserted">+       res.redirect("/api/public");</span><span class="token inserted">+     });</span><span class="token inserted">+   });</span><span class="token inserted">+ });</span>  export default router;</code></pre><p>Please note the <code>express.urlencoded</code> middleware added to <code>/login/callback</code>. If this is skipped, you’ll face infinite login loops.</p><p>Related posts:</p><ul><li><a href="https://www.sheshbabu.com/posts/visual-explanation-of-saml-authentication/">Visual explanation of SAML authentication</a></li><li><a href="https://www.sheshbabu.com/posts/running-express-over-https-in-localhost/">Running Express over HTTPS in localhost</a></li></ul>]]></content>
    
    <summary type="html">
    
      Authenticate with Microsoft Entra ID (Azure Active Directory) using Node.js, Express and Passport
    
    </summary>
    
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
      <category term="Node.js" scheme="https://www.sheshbabu.com/tags/Node-js/"/>
    
      <category term="SAML" scheme="https://www.sheshbabu.com/tags/SAML/"/>
    
      <category term="Authentication" scheme="https://www.sheshbabu.com/tags/Authentication/"/>
    
  </entry>
  
  <entry>
    <title>Refreshing OAuth Access Tokens using Axios Interceptors</title>
    <link href="https://www.sheshbabu.com/posts/refreshing-oauth-access-tokens-using-axios-interceptors/"/>
    <id>https://www.sheshbabu.com/posts/refreshing-oauth-access-tokens-using-axios-interceptors/</id>
    <published>2024-08-11T01:03:50.000Z</published>
    <updated>2024-08-11T03:12:37.370Z</updated>
    
    <content type="html"><![CDATA[<p>HTTP Clients like <a href="https://axios-http.com/docs/interceptors" target="_blank" rel="noopener">Axios</a>, <a href="https://www.python-httpx.org/advanced/event-hooks/" target="_blank" rel="noopener">HTTPX</a>, etc, allow you to hook into the HTTP request lifecycle so you can add custom logic before sending a request or after receiving a response. These “hooks” or “interceptors” are useful for things like adding logs, propagating distributed tracing IDs etc. </p><p>Interceptors can also be used to refresh OAuth Access Tokens. Access Tokens are used to authenticate the client application, and are sent in HTTP requests to the resource server. Access Tokens are typically short-lived for better security, and when they expire, new tokens can be generated by the authorization server.</p><p>For this post, we’ll be working with the <a href="https://learn.microsoft.com/en-us/graph/auth-v2-service" target="_blank" rel="noopener">Microsoft Graph</a> API and using the <a href="https://auth0.com/docs/get-started/authentication-and-authorization-flow/client-credentials-flow" target="_blank" rel="noopener">client credentials</a> flow, but the same logic can be applied for other endpoints as well.</p><p>We’ll be talking to two servers:</p><ol><li><strong>Resource Server:</strong> The endpoints we want to access.</li><li><strong>Token Server:</strong> The server which gives us Access Tokens to authenticate with the Resource Server.</li></ol><p>Let’s start by creating the <a href="https://www.sheshbabu.com/posts/organizing-http-requests-using-api-module-pattern/">API client module</a> for Microsoft Graph.</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// ./ApiClient.js</span><span class="token keyword">import</span> axios <span class="token keyword">from</span> <span class="token string">"axios"</span><span class="token punctuation">;</span><span class="token keyword">const</span> TOKEN_URL <span class="token operator">=</span> <span class="token string">"https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token"</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// replace 'tenant' before using</span><span class="token keyword">const</span> RESOURCE_BASE_URL <span class="token operator">=</span> <span class="token string">"https://graph.microsoft.com/v1.0"</span><span class="token punctuation">;</span><span class="token keyword">const</span> ApiClient <span class="token operator">=</span> axios<span class="token punctuation">.</span><span class="token function">create</span><span class="token punctuation">(</span><span class="token punctuation">{</span>    baseURL<span class="token punctuation">:</span> RESOURCE_BASE_URL<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>ApiClient<span class="token punctuation">.</span>interceptors<span class="token punctuation">.</span>request<span class="token punctuation">.</span><span class="token function">use</span><span class="token punctuation">(</span>injectAccessToken<span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">injectAccessToken</span><span class="token punctuation">(</span>config<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">return</span> <span class="token punctuation">{</span>        <span class="token operator">...</span>config    <span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">export</span> <span class="token keyword">default</span> ApiClient<span class="token punctuation">;</span></code></pre><p>We’ve left the <code>injectAccessToken</code> function as a no-op for now. This API client module can be used in other modules to make calls to resource server endpoints. For example, let’s call the <a href="https://learn.microsoft.com/en-us/graph/auth-v2-service?tabs=http#5-use-the-access-token-to-call-microsoft-graph" target="_blank" rel="noopener">get all users endpoint</a> in Microsoft Graph:</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// ./index.js</span><span class="token keyword">import</span> ApiClient <span class="token keyword">from</span> <span class="token string">'./ApiClient.js'</span><span class="token punctuation">;</span><span class="token keyword">const</span> response <span class="token operator">=</span> <span class="token keyword">await</span> ApiClient<span class="token punctuation">.</span><span class="token keyword">get</span><span class="token punctuation">(</span><span class="token string">"/users"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>response<span class="token punctuation">.</span>data<span class="token punctuation">.</span>value<span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>Let’s flesh out the <code>injectAccessToken</code> function. We want this function to request Access Token and store it somewhere. If the Access Token has not expired, we use it in all the requests to Resource Server. If it has expired, we fetch new token and update the store. For token storage, we can just create a module-level variable called <code>token</code> as it would be overkill to use a database.</p><p>Here’s the high-level flow as code:</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// ./ApiClient.js</span><span class="token keyword">let</span> token <span class="token operator">=</span> <span class="token string">""</span><span class="token punctuation">;</span><span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">injectAccessToken</span><span class="token punctuation">(</span>config<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token comment" spellcheck="true">// Don't use the interceptor for TOKEN_URL</span>  <span class="token keyword">if</span> <span class="token punctuation">(</span>config<span class="token punctuation">.</span>url <span class="token operator">===</span> TOKEN_URL<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">return</span> config<span class="token punctuation">;</span>  <span class="token punctuation">}</span>  <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">hasAccessTokenExpired</span><span class="token punctuation">(</span>token<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    token <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">acquireAccessToken</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span>  <span class="token keyword">const</span> headers <span class="token operator">=</span> config<span class="token punctuation">.</span>headers<span class="token punctuation">.</span><span class="token function">concat</span><span class="token punctuation">(</span><span class="token punctuation">{</span>Authorization<span class="token punctuation">:</span> <span class="token template-string"><span class="token string">`Bearer </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>token<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">`</span></span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">return</span> <span class="token punctuation">{</span>    <span class="token operator">...</span>config<span class="token punctuation">,</span>    headers<span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>Microsoft Graph uses <a href="https://www.oauth.com/oauth2-servers/access-tokens/self-encoded-access-tokens/" target="_blank" rel="noopener">JWT as Access Tokens</a>, so we can inspect the payload to check token expiry:</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// ./ApiClient.js</span><span class="token keyword">import</span> <span class="token punctuation">{</span> jwtDecode <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'jwt-decode'</span><span class="token punctuation">;</span><span class="token keyword">function</span> <span class="token function">hasAccessTokenExpired</span><span class="token punctuation">(</span>token<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">if</span> <span class="token punctuation">(</span>token <span class="token operator">===</span> <span class="token string">""</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">return</span> <span class="token boolean">true</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span>  <span class="token keyword">const</span> payload <span class="token operator">=</span> <span class="token function">jwtDecode</span><span class="token punctuation">(</span>token<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">return</span> Date<span class="token punctuation">.</span><span class="token function">now</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">>=</span> payload<span class="token punctuation">.</span>exp <span class="token operator">*</span> <span class="token number">1000</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>We can follow the Microsoft Graph <a href="https://learn.microsoft.com/en-us/graph/auth-v2-service?tabs=http#token-request" target="_blank" rel="noopener">docs for making a token request</a>:</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// ./ApiClient.js</span><span class="token keyword">import</span> qs <span class="token keyword">from</span> <span class="token string">'qs'</span><span class="token punctuation">;</span><span class="token keyword">const</span> CLIENT_ID <span class="token operator">=</span> <span class="token string">""</span><span class="token punctuation">;</span><span class="token keyword">const</span> CLIENT_SECRET <span class="token operator">=</span> <span class="token string">""</span><span class="token punctuation">;</span><span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">acquireAccessToken</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">const</span> config <span class="token operator">=</span> <span class="token punctuation">{</span>    headers<span class="token punctuation">:</span> <span class="token punctuation">{</span> <span class="token string">"Content-Type"</span><span class="token punctuation">:</span> <span class="token string">"application/x-www-form-urlencoded"</span> <span class="token punctuation">}</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">;</span>  <span class="token keyword">const</span> body <span class="token operator">=</span> qs<span class="token punctuation">.</span><span class="token function">stringify</span><span class="token punctuation">(</span><span class="token punctuation">{</span>    client_id<span class="token punctuation">:</span> CLIENT_ID<span class="token punctuation">,</span>    client_secret<span class="token punctuation">:</span> CLIENT_SECRET<span class="token punctuation">,</span>    grant_type<span class="token punctuation">:</span> <span class="token string">"client_credentials"</span><span class="token punctuation">,</span>    scope<span class="token punctuation">:</span> <span class="token string">"https://graph.microsoft.com/.default"</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">const</span> response <span class="token operator">=</span> <span class="token keyword">await</span> axios<span class="token punctuation">.</span><span class="token function">post</span><span class="token punctuation">(</span>TOKEN_URL<span class="token punctuation">,</span> body<span class="token punctuation">,</span> config<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">return</span> response<span class="token punctuation">.</span>data<span class="token punctuation">.</span>access_token<span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>Putting them all together:</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// ./ApiClient.js</span><span class="token keyword">import</span> axios <span class="token keyword">from</span> <span class="token string">"axios"</span><span class="token punctuation">;</span><span class="token keyword">import</span> <span class="token punctuation">{</span> jwtDecode <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"jwt-decode"</span><span class="token punctuation">;</span><span class="token keyword">import</span> qs <span class="token keyword">from</span> <span class="token string">"qs"</span><span class="token punctuation">;</span><span class="token comment" spellcheck="true">// Store these in environment variables</span><span class="token keyword">const</span> TOKEN_URL <span class="token operator">=</span> <span class="token string">"https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token"</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// replace 'tenant' before using</span><span class="token keyword">const</span> RESOURCE_BASE_URL <span class="token operator">=</span> <span class="token string">"https://graph.microsoft.com/v1.0"</span><span class="token punctuation">;</span><span class="token keyword">const</span> CLIENT_ID <span class="token operator">=</span> <span class="token string">""</span><span class="token punctuation">;</span><span class="token keyword">const</span> CLIENT_SECRET <span class="token operator">=</span> <span class="token string">""</span><span class="token punctuation">;</span><span class="token keyword">let</span> token <span class="token operator">=</span> <span class="token string">""</span><span class="token punctuation">;</span><span class="token keyword">const</span> ApiClient <span class="token operator">=</span> axios<span class="token punctuation">.</span><span class="token function">create</span><span class="token punctuation">(</span><span class="token punctuation">{</span>  baseURL<span class="token punctuation">:</span> RESOURCE_BASE_URL<span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>ApiClient<span class="token punctuation">.</span>interceptors<span class="token punctuation">.</span>request<span class="token punctuation">.</span><span class="token function">use</span><span class="token punctuation">(</span>injectAccessToken<span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">injectAccessToken</span><span class="token punctuation">(</span>config<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token comment" spellcheck="true">// Don't use the interceptor for TOKEN_URL</span>  <span class="token keyword">if</span> <span class="token punctuation">(</span>config<span class="token punctuation">.</span>url <span class="token operator">===</span> TOKEN_URL<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">return</span> config<span class="token punctuation">;</span>  <span class="token punctuation">}</span>  <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">hasAccessTokenExpired</span><span class="token punctuation">(</span>token<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    token <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">acquireAccessToken</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span>  <span class="token keyword">const</span> headers <span class="token operator">=</span> config<span class="token punctuation">.</span>headers<span class="token punctuation">.</span><span class="token function">concat</span><span class="token punctuation">(</span><span class="token punctuation">{</span> Authorization<span class="token punctuation">:</span> <span class="token template-string"><span class="token string">`Bearer </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>token<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">`</span></span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">return</span> <span class="token punctuation">{</span>    <span class="token operator">...</span>config<span class="token punctuation">,</span>    headers<span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">function</span> <span class="token function">hasAccessTokenExpired</span><span class="token punctuation">(</span>token<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">if</span> <span class="token punctuation">(</span>token <span class="token operator">===</span> <span class="token string">""</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">return</span> <span class="token boolean">true</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span>  <span class="token keyword">const</span> payload <span class="token operator">=</span> <span class="token function">jwtDecode</span><span class="token punctuation">(</span>token<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">return</span> Date<span class="token punctuation">.</span><span class="token function">now</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">>=</span> payload<span class="token punctuation">.</span>exp <span class="token operator">*</span> <span class="token number">1000</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">acquireAccessToken</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">const</span> config <span class="token operator">=</span> <span class="token punctuation">{</span>    headers<span class="token punctuation">:</span> <span class="token punctuation">{</span> <span class="token string">"Content-Type"</span><span class="token punctuation">:</span> <span class="token string">"application/x-www-form-urlencoded"</span> <span class="token punctuation">}</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">;</span>  <span class="token keyword">const</span> body <span class="token operator">=</span> qs<span class="token punctuation">.</span><span class="token function">stringify</span><span class="token punctuation">(</span><span class="token punctuation">{</span>    client_id<span class="token punctuation">:</span> CLIENT_ID<span class="token punctuation">,</span>    client_secret<span class="token punctuation">:</span> CLIENT_SECRET<span class="token punctuation">,</span>    grant_type<span class="token punctuation">:</span> <span class="token string">"client_credentials"</span><span class="token punctuation">,</span>    scope<span class="token punctuation">:</span> <span class="token string">"https://graph.microsoft.com/.default"</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">const</span> response <span class="token operator">=</span> <span class="token keyword">await</span> axios<span class="token punctuation">.</span><span class="token function">post</span><span class="token punctuation">(</span>TOKEN_URL<span class="token punctuation">,</span> body<span class="token punctuation">,</span> config<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">return</span> response<span class="token punctuation">.</span>data<span class="token punctuation">.</span>access_token<span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">export</span> <span class="token keyword">default</span> ApiClient<span class="token punctuation">;</span></code></pre>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;HTTP Clients like &lt;a href=&quot;https://axios-http.com/docs/interceptors&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Axios&lt;/a&gt;, &lt;a href=&quot;https://www.pytho
      
    
    </summary>
    
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
      <category term="Node.js" scheme="https://www.sheshbabu.com/tags/Node-js/"/>
    
      <category term="Authentication" scheme="https://www.sheshbabu.com/tags/Authentication/"/>
    
      <category term="Microsoft Graph" scheme="https://www.sheshbabu.com/tags/Microsoft-Graph/"/>
    
  </entry>
  
  <entry>
    <title>FastAPI without ORM: Testing with asyncpg</title>
    <link href="https://www.sheshbabu.com/posts/fastapi-without-orm-testing-with-asyncpg/"/>
    <id>https://www.sheshbabu.com/posts/fastapi-without-orm-testing-with-asyncpg/</id>
    <published>2024-08-10T02:04:59.000Z</published>
    <updated>2024-08-10T06:31:00.098Z</updated>
    
    <content type="html"><![CDATA[<p>In our last two posts, we explored how to <a href="https://www.sheshbabu.com/posts/fastapi-without-orm-getting-started-with-asyncpg/">setup asyncpg with FastAPI</a> and how to write <a href="https://www.sheshbabu.com/posts/demystifying-postgres-schema-migrations/">database schema migrations</a>. </p><p>We’ll write end-to-end integration tests that will exercise the code from API endpoints through business logic to database queries. We will create tests for two endpoints which can then be extended to more endpoints. We’ll be following the <a href="https://github.com/sheshbabu/fastapi-asyncpg-demo" target="_blank" rel="noopener">same codebase</a> from the previous two posts.</p><h3 id="Setup-and-Teardown"><a href="#Setup-and-Teardown" class="headerlink" title="Setup and Teardown"></a>Setup and Teardown</h3><p>To ensure that the tests are independent from each other, we must reset the database and apply migrations before and after each test. We can use <code>fixtures</code> in pytest to implement this in a clean manner:</p><pre class=" language-python"><code class="language-python"><span class="token keyword">import</span> pytest<span class="token keyword">from</span> src<span class="token punctuation">.</span>commons<span class="token punctuation">.</span>postgres <span class="token keyword">import</span> database<span class="token keyword">from</span> src<span class="token punctuation">.</span>commons <span class="token keyword">import</span> migrate@pytest<span class="token punctuation">.</span>fixture<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">setup_database</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>    <span class="token keyword">await</span> database<span class="token punctuation">.</span>connect<span class="token punctuation">(</span><span class="token punctuation">)</span>    <span class="token keyword">async</span> <span class="token keyword">with</span> database<span class="token punctuation">.</span>pool<span class="token punctuation">.</span>acquire<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token keyword">as</span> connection<span class="token punctuation">:</span>        <span class="token keyword">await</span> connection<span class="token punctuation">.</span>execute<span class="token punctuation">(</span><span class="token string">"CREATE SCHEMA IF NOT EXISTS public;"</span><span class="token punctuation">)</span>    <span class="token keyword">await</span> migrate<span class="token punctuation">.</span>apply_pending_migrations<span class="token punctuation">(</span><span class="token punctuation">)</span>    <span class="token keyword">yield</span>    <span class="token keyword">async</span> <span class="token keyword">with</span> database<span class="token punctuation">.</span>pool<span class="token punctuation">.</span>acquire<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token keyword">as</span> connection<span class="token punctuation">:</span>        <span class="token keyword">await</span> connection<span class="token punctuation">.</span>execute<span class="token punctuation">(</span><span class="token string">"DROP SCHEMA IF EXISTS public CASCADE;"</span><span class="token punctuation">)</span>    <span class="token keyword">await</span> database<span class="token punctuation">.</span>disconnect<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre><h3 id="Testing-GET-users-endpoint"><a href="#Testing-GET-users-endpoint" class="headerlink" title="Testing GET /users endpoint"></a>Testing GET /users endpoint</h3><p>Now that we’ve the setup/teardown in place, let’s start writing the first set of tests. We’ll focus on the <a href="https://github.com/sheshbabu/fastapi-asyncpg-demo/blob/master/src/users/users_route.py" target="_blank" rel="noopener"><code>GET /users</code></a> endpoint.</p><pre class=" language-python"><code class="language-python"><span class="token keyword">from</span> httpx <span class="token keyword">import</span> AsyncClient<span class="token punctuation">,</span> ASGITransport<span class="token keyword">from</span> src<span class="token punctuation">.</span>main <span class="token keyword">import</span> app@pytest<span class="token punctuation">.</span>fixture<span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">client_app</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>    <span class="token keyword">async</span> <span class="token keyword">with</span> AsyncClient<span class="token punctuation">(</span>transport<span class="token operator">=</span>ASGITransport<span class="token punctuation">(</span>app<span class="token operator">=</span>app<span class="token punctuation">)</span><span class="token punctuation">,</span> base_url<span class="token operator">=</span><span class="token string">"http://test"</span><span class="token punctuation">)</span> <span class="token keyword">as</span> client<span class="token punctuation">:</span>        <span class="token keyword">yield</span> client@pytest<span class="token punctuation">.</span>mark<span class="token punctuation">.</span>asyncio<span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">test_get_users_returns_empty_list_when_no_users_exist</span><span class="token punctuation">(</span>setup_database<span class="token punctuation">,</span> client_app<span class="token punctuation">)</span><span class="token punctuation">:</span>    response <span class="token operator">=</span> <span class="token keyword">await</span> client_app<span class="token punctuation">.</span>get<span class="token punctuation">(</span><span class="token string">"/users/"</span><span class="token punctuation">)</span>    <span class="token keyword">assert</span> response<span class="token punctuation">.</span>status_code <span class="token operator">==</span> <span class="token number">200</span>    <span class="token keyword">assert</span> response<span class="token punctuation">.</span>json<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">==</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>@pytest<span class="token punctuation">.</span>mark<span class="token punctuation">.</span>asyncio<span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">test_get_users_returns_users_list_when_users_exist</span><span class="token punctuation">(</span>setup_database<span class="token punctuation">,</span> client_app<span class="token punctuation">)</span><span class="token punctuation">:</span>    <span class="token keyword">async</span> <span class="token keyword">with</span> database<span class="token punctuation">.</span>pool<span class="token punctuation">.</span>acquire<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token keyword">as</span> connection<span class="token punctuation">:</span>        <span class="token keyword">await</span> connection<span class="token punctuation">.</span>execute<span class="token punctuation">(</span><span class="token string">"INSERT INTO users (name, email) VALUES ($1, $2)"</span><span class="token punctuation">,</span> <span class="token string">"Shesh"</span><span class="token punctuation">,</span> <span class="token string">"sheshbabu@gmail.com"</span><span class="token punctuation">)</span>    response <span class="token operator">=</span> <span class="token keyword">await</span> client_app<span class="token punctuation">.</span>get<span class="token punctuation">(</span><span class="token string">"/users/"</span><span class="token punctuation">)</span>    <span class="token keyword">assert</span> response<span class="token punctuation">.</span>status_code <span class="token operator">==</span> <span class="token number">200</span>    <span class="token keyword">assert</span> response<span class="token punctuation">.</span>json<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">==</span> <span class="token punctuation">[</span><span class="token punctuation">{</span><span class="token string">"email"</span><span class="token punctuation">:</span> <span class="token string">"sheshbabu@gmail.com"</span><span class="token punctuation">,</span> <span class="token string">"name"</span><span class="token punctuation">:</span> <span class="token string">"Shesh"</span><span class="token punctuation">}</span><span class="token punctuation">]</span></code></pre><p>Here, we added two scenarios - one with an empty table and another with a single record. We’ve used the <code>app</code> object to make the calls. This exercises the code paths all the way to database and back.</p><h3 id="Testing-POST-users-endpoint"><a href="#Testing-POST-users-endpoint" class="headerlink" title="Testing POST /users endpoint"></a>Testing POST /users endpoint</h3><p>We’ll go through one more example so we know how the code is organized.</p><pre class=" language-python"><code class="language-python">@pytest<span class="token punctuation">.</span>mark<span class="token punctuation">.</span>asyncio<span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">test_post_users_returns_success_status_when_valid_payload_is_provided</span><span class="token punctuation">(</span>setup_database<span class="token punctuation">,</span> client_app<span class="token punctuation">)</span><span class="token punctuation">:</span>    payload <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">"name"</span><span class="token punctuation">:</span> <span class="token string">"Shesh"</span><span class="token punctuation">,</span> <span class="token string">"email"</span><span class="token punctuation">:</span> <span class="token string">"sheshbabu@gmail.com"</span><span class="token punctuation">}</span>    response <span class="token operator">=</span> <span class="token keyword">await</span> client_app<span class="token punctuation">.</span>post<span class="token punctuation">(</span><span class="token string">"/users/"</span><span class="token punctuation">,</span> json<span class="token operator">=</span>payload<span class="token punctuation">)</span>    <span class="token keyword">assert</span> response<span class="token punctuation">.</span>status_code <span class="token operator">==</span> <span class="token number">200</span></code></pre><p>That’s it! We can now use this approach to add more test cases for this app.</p><p>Related posts:</p><ul><li><a href="https://www.sheshbabu.com/posts/fastapi-without-orm-getting-started-with-asyncpg/">FastAPI without ORM: Getting started with asyncpg</a></li><li><a href="https://www.sheshbabu.com/posts/demystifying-postgres-schema-migrations/">Demystifying Postgres Schema Migrations: Building it from Scratch</a></li><li><a href="https://www.sheshbabu.com/posts/fastapi-structured-json-logging/">Structured JSON Logging using FastAPI</a></li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;In our last two posts, we explored how to &lt;a href=&quot;https://www.sheshbabu.com/posts/fastapi-without-orm-getting-started-with-asyncpg/&quot;&gt;set
      
    
    </summary>
    
    
      <category term="Python" scheme="https://www.sheshbabu.com/tags/Python/"/>
    
      <category term="Testing" scheme="https://www.sheshbabu.com/tags/Testing/"/>
    
      <category term="Postgres" scheme="https://www.sheshbabu.com/tags/Postgres/"/>
    
      <category term="asyncpg" scheme="https://www.sheshbabu.com/tags/asyncpg/"/>
    
      <category term="FastAPI" scheme="https://www.sheshbabu.com/tags/FastAPI/"/>
    
  </entry>
  
  <entry>
    <title>System Design for Entity Resolution</title>
    <link href="https://www.sheshbabu.com/posts/system-design-for-entity-resolution/"/>
    <id>https://www.sheshbabu.com/posts/system-design-for-entity-resolution/</id>
    <published>2024-08-08T11:13:13.000Z</published>
    <updated>2024-08-09T13:06:12.273Z</updated>
    
    <content type="html"><![CDATA[<p>Entity Resolution (also known as Record Linkage, Data Fusion, Deduplication etc) is the process of identifying the same real-world entities (person, company, product etc) from two or more different datasets and combining them into a unique “golden record”. This golden record would serve as an authoritative source of truth for all entities in your organization. </p><p>This post aims to serve as a guide for those planning to implement an entity resolution system inside your organization. As mentioned in my <a href="https://www.sheshbabu.com/posts/entity-resolution-challenges/">previous post</a>, every entity resolution project presents unique challenges based on cleanliness and scale of the data sources, so your design might look different from this and that’s totally fine. Even if you’re thinking of buying an off-the-shelf solution, it’s still useful to know the different aspects of such systems so you can make an informed purchasing decision.</p><p><img src="/images/2024-system-design-for-entity-resolution/2024-system-design-for-entity-resolution.png" alt=""></p><h2 id="High-level-stages"><a href="#High-level-stages" class="headerlink" title="High level stages"></a>High level stages</h2><p>Entity Resolution typically has the following stages:</p><ol><li>Combine</li><li>Blocking</li><li>Linkage</li><li>Clustering</li></ol><h2 id="Combine"><a href="#Combine" class="headerlink" title="Combine"></a>Combine</h2><p>The first step is to ingest and consolidate the data from different places like S3, Excel, SQL database etc into a single unified dataset. </p><h3 id="Schema-Matching"><a href="#Schema-Matching" class="headerlink" title="Schema Matching"></a>Schema Matching</h3><p>In this step, we need to select which attributes across the different datasets have the same information. These selected attributes would help in the downstream linkage process and would finally end up as attributes of the final golden record.</p><h3 id="Validation"><a href="#Validation" class="headerlink" title="Validation"></a>Validation</h3><p>Depending on your data sources, you might need to implement certain validation rules to make sure that clean data is passed on to downstream stages. These might be about ensuring the email addresses are valid, phone numbers have the correct length etc. If the attributes are incorrect, depending on your use case, you can decide to either convert them to NULL or skip ingesting the record altogether.</p><h3 id="Normalization"><a href="#Normalization" class="headerlink" title="Normalization"></a>Normalization</h3><p>There’s a good chance different data sources have the same attribute captured in different formats. For example, one data source might collect date of birth as “01 Jan 1900” and other might collect that as “1900/01/01”. So in this step, you’ll need to normalize/standardize these into the same formats so they’re more useful in downstream stages. It’s helpful to maintain both “raw” and “clean” versions of the attributes for future troubleshooting.</p><h3 id="Deduplication"><a href="#Deduplication" class="headerlink" title="Deduplication"></a>Deduplication</h3><p>Some data sources have duplicate records for the same entity. It’s better to clean these up earlier in the process before you merge this data source with other data sources. “Deduplication” is a special case of record linkage where the linkages happen within a single dataset as opposed to happening across multiple datasets.</p><h2 id="Blocking"><a href="#Blocking" class="headerlink" title="Blocking"></a>Blocking</h2><p>One of the challenges of entity resolution is that the data is huge. We need to do a pairwise comparison between the records of the two datasets and decide whether they match or not. This pairwise comparison scales quadratically - if you have <code>m</code> number of records in one dataset and <code>n</code> number of records in the other, then the number of comparisons is <code>m * n</code>. If you remove duplicates, it becomes <code>(m * n) / 2</code>.</p><p><img src="/images/2023-entity-resolution-challenges/2023-entity-resolution-challenges-01.png" alt=""></p><p>In order to do this efficiently, we need to reduce the search space. If you think about this, it doesn’t make sense to compare every single record. We need to figure out a way to group the records that are likely to be a match. This step is called “blocking” or “indexing”.</p><h2 id="Linkage"><a href="#Linkage" class="headerlink" title="Linkage"></a>Linkage</h2><p>In this stage, we make a decision to link a record from one data source to its identical record in another data source. It’s useful to <a href="https://www.sheshbabu.com/posts/graph-retrieval-using-postgres-recursive-ctes/">store these links as a graph</a> as linkages are usually transitive in nature.</p><h3 id="Deterministic-Linkage"><a href="#Deterministic-Linkage" class="headerlink" title="Deterministic Linkage"></a>Deterministic Linkage</h3><p>This is a rules-based approach. If two records have one or more attributes with identical values, then they’re classified as a match and linked together.</p><ul><li><strong>Direct Deterministic Linkage:</strong> The attributes used are unique identifiers like SSN, email etc. </li><li><strong>Exact/Strict Deterministic Linkage:</strong> The attributes must be exactly the same in the record pair.</li><li><strong>Approximate Deterministic Linkage:</strong> The attributes can have minor differences between two records.</li><li><strong>Stepwise Deterministic Linkage:</strong> The attributes are matched in a series of less restrictive steps.</li></ul><p>Deterministic linkage is the oldest, simplest and computationally fastest approach. It avoids false matches and provides acceptable results. The downside is that it misses out a lot of matches because of recording errors, missing values, variations, changes over time, etc.</p><h3 id="Probabilistic-Linkage"><a href="#Probabilistic-Linkage" class="headerlink" title="Probabilistic Linkage"></a>Probabilistic Linkage</h3><p>If unique identifiers are unavailable, then we need to use an evidence-based approach. The likelihood of two records referring to the same entity increases if they share the same attribute values. The probability increases as more attributes match, decreases if there’s a mismatch.</p><h3 id="ML-based-Linkage"><a href="#ML-based-Linkage" class="headerlink" title="ML-based Linkage"></a>ML-based Linkage</h3><p>If we’re able to provide labelled data for which record pairs match or don’t match, then a ML model can be trained to do the classification.</p><h2 id="Clustering"><a href="#Clustering" class="headerlink" title="Clustering"></a>Clustering</h2><p>Once we’re done with the linkage process, we can see <a href="https://www.sheshbabu.com/posts/detecting-clusters-in-graphs-using-networkx/">clusters forming in the graph</a>. Each of these cluster is a unique entity across multiple data sources. A golden record is created from each cluster by giving it an unique identifier and merging the attributes from records originating from different sources. If there’s a conflict in attribute value, we can use a data source prioritization list to decide which data source to pick that value from. It’s helpful to keep the mapping between this golden record id and the ids of records from different data sources.</p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>Whether you’re building this in-house or buying an off-the-shelf solution, it’s important to have a decent understanding of the data sources you’re working with. It’s better to do an <a href="https://en.wikipedia.org/wiki/Exploratory_data_analysis" target="_blank" rel="noopener">EDA</a> on each data source to understand the format of the attribute values, whether the values are valid, are there any duplicate entities within that data source etc. It’s also advisable to consult with a domain expert before starting the implementation. Entity resolution involves a lot of defensive programming, so the more you know about the data sources, the more surprises you can avoid in future.</p><h2 id="References"><a href="#References" class="headerlink" title="References"></a>References</h2><ul><li><a href="https://www.ncbi.nlm.nih.gov/books/NBK253312/" target="_blank" rel="noopener">Linking Data for Health Services Research</a></li><li><a href="https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-020-01285-w" target="_blank" rel="noopener">CIDACS-RL: a novel indexing search and scoring-based record linkage system for huge datasets with high accuracy and scalability</a></li><li><a href="https://www.science.org/doi/10.1126/science.130.3381.954" target="_blank" rel="noopener">Automatic Linkage of Vital Records</a></li><li><a href="https://courses.cs.washington.edu/courses/cse590q/04au/papers/Felligi69.pdf" target="_blank" rel="noopener">A Theory for Record Linkage</a></li><li><a href="https://www.robinlinacre.com/intro_to_probabilistic_linkage/" target="_blank" rel="noopener">An Interactive Introduction to the Fellegi-Sunter Model for Data Linkage/Deduplication</a></li><li><a href="http://users.cecs.anu.edu.au/~Peter.Christen/publications/christen2011indexing.pdf" target="_blank" rel="noopener">A Survey of Indexing Techniques for Scalable Record Linkage and Deduplication</a></li><li><a href="https://arxiv.org/abs/1905.06167" target="_blank" rel="noopener">A Survey of Blocking and Filtering Techniques for Entity Resolution</a></li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;Entity Resolution (also known as Record Linkage, Data Fusion, Deduplication etc) is the process of identifying the same real-world entiti
      
    
    </summary>
    
    
      <category term="Entity Resolution" scheme="https://www.sheshbabu.com/tags/Entity-Resolution/"/>
    
      <category term="System Design" scheme="https://www.sheshbabu.com/tags/System-Design/"/>
    
  </entry>
  
  <entry>
    <title>Demystifying Postgres Schema Migrations: Building it from Scratch</title>
    <link href="https://www.sheshbabu.com/posts/demystifying-postgres-schema-migrations/"/>
    <id>https://www.sheshbabu.com/posts/demystifying-postgres-schema-migrations/</id>
    <published>2024-08-07T13:24:45.000Z</published>
    <updated>2024-08-07T13:02:49.671Z</updated>
    
    <content type="html"><![CDATA[<p>Database schema migrations are rarely taught in universities, and if you’re a self-taught developer, migrations are not something you would naturally pick up.</p><p>In this post, we’ll learn about what issues they solve and how they work. We’ll also build a simple implementation. I believe the best way to learn something is to build it from scratch. Even if you end up using a more mature library, by building from scratch you’ll learn about the internal concepts which will help you a lot in troubleshooting. </p><p><img src="/images/2024-demystifying-postgres-schema-migrations/2024-demystifying-postgres-schema-migrations-01.gif" alt=""></p><h2 id="Why-do-we-need-this"><a href="#Why-do-we-need-this" class="headerlink" title="Why do we need this?"></a>Why do we need this?</h2><p>If you work with databases but are unfamiliar with migrations, how do you make schema changes like adding new tables, altering columns, etc? There’s a good chance you directly execute SQL commands in the database when the schema changes. </p><p>If this works, then what’s wrong with this approach? </p><ul><li>Directly executing SQL commands is a manual process and it’s easy to make mistakes. </li><li>If you have multiple environments like dev, staging, and prod, there’s a chance some of the steps might get missed out.</li><li>There’s no record of what changes were applied and in which sequence.</li><li>If two people on the team want to make changes to the schema, there’s a chance of conflicts.</li></ul><h2 id="How-do-migrations-solve-these-issues"><a href="#How-do-migrations-solve-these-issues" class="headerlink" title="How do migrations solve these issues?"></a>How do migrations solve these issues?</h2><p>Migrations are written as plaintext files, usually in SQL or using ORM methods. These files are stored along with the application code in version control systems like Git. When an application is being deployed, the CI/CD pipeline or the application startup code executes the migration files and then proceeds with running the application.</p><p>As the migrations are stored in version control, they can be deterministically executed in multiple environments. Code reviews can be used to notify other developers of changes and to seek feedback.</p><h2 id="Building-it-from-scratch"><a href="#Building-it-from-scratch" class="headerlink" title="Building it from scratch"></a>Building it from scratch</h2><p>Now that we know why they’re important, let’s learn how they work. The best way to learn something is to build a simple version from scratch. </p><p>Based on the above, we need the following:</p><ol><li>Every time there’s a schema change, we write the change as an SQL file.</li><li>There should be a way for these files to indicate the order in which they need to be executed.</li><li>The changes that are previously applied are tracked somewhere so they’re not applied again.</li></ol><p>Points 1 &amp; 2 can be achieved by creating an SQL file for each schema change and prefixing the file names with an auto-incrementing integer or timestamps:</p><pre class=" language-shell"><code class="language-shell">src/`-- migrations    |-- 001_init_database.sql    |-- 002_create_users_table.sql    |-- 003_add_email_to_users_table.sql    |-- 004_create_audit_table.sql    `-- 005_create_users_index.sql</code></pre><p>Point 3 can be solved by creating a table <code>schema_migrations</code> in the database to keep track of all the applied migrations:</p><pre class=" language-sql"><code class="language-sql"><span class="token keyword">CREATE</span> <span class="token keyword">TABLE</span> <span class="token keyword">IF</span> <span class="token operator">NOT</span> <span class="token keyword">EXISTS</span> schema_migrations <span class="token punctuation">(</span>    version     <span class="token keyword">INTEGER</span>   <span class="token keyword">PRIMARY</span> <span class="token keyword">KEY</span><span class="token punctuation">,</span>    name        <span class="token keyword">TEXT</span>      <span class="token operator">NOT</span> <span class="token boolean">NULL</span><span class="token punctuation">,</span>    migrated_at <span class="token keyword">TIMESTAMP</span> <span class="token operator">NOT</span> <span class="token boolean">NULL</span> <span class="token keyword">DEFAULT</span> <span class="token function">NOW</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><pre class=" language-sql"><code class="language-sql"><span class="token operator">+</span><span class="token comment" spellcheck="true">-------------+----------------------------------+----------------------------+</span><span class="token operator">|</span>   version   <span class="token operator">|</span>               name               <span class="token operator">|</span>         migrated_at        <span class="token operator">|</span><span class="token operator">+</span><span class="token comment" spellcheck="true">-------------+----------------------------------+----------------------------+</span><span class="token operator">|</span>     <span class="token number">1</span>       <span class="token operator">|</span> 001_init_database                <span class="token operator">|</span> <span class="token number">2024</span><span class="token operator">-</span><span class="token number">08</span><span class="token operator">-</span><span class="token number">06</span> <span class="token number">10</span>:<span class="token number">15</span>:<span class="token number">30.123456</span> <span class="token operator">|</span><span class="token operator">|</span>     <span class="token number">2</span>       <span class="token operator">|</span> 002_create_users_table           <span class="token operator">|</span> <span class="token number">2024</span><span class="token operator">-</span><span class="token number">08</span><span class="token operator">-</span><span class="token number">06</span> <span class="token number">11</span>:<span class="token number">30</span>:<span class="token number">45.789012</span> <span class="token operator">|</span><span class="token operator">|</span>     <span class="token number">3</span>       <span class="token operator">|</span> 003_add_email_to_users_table     <span class="token operator">|</span> <span class="token number">2024</span><span class="token operator">-</span><span class="token number">08</span><span class="token operator">-</span><span class="token number">07</span> <span class="token number">09</span>:<span class="token number">20</span>:<span class="token number">15.456789</span> <span class="token operator">|</span><span class="token operator">|</span>     <span class="token number">4</span>       <span class="token operator">|</span> 004_create_audit_table           <span class="token operator">|</span> <span class="token number">2024</span><span class="token operator">-</span><span class="token number">08</span><span class="token operator">-</span><span class="token number">07</span> <span class="token number">10</span>:<span class="token number">05</span>:<span class="token number">00.234567</span> <span class="token operator">|</span><span class="token operator">|</span>     <span class="token number">5</span>       <span class="token operator">|</span> 005_create_users_index           <span class="token operator">|</span> <span class="token number">2024</span><span class="token operator">-</span><span class="token number">08</span><span class="token operator">-</span><span class="token number">08</span> <span class="token number">14</span>:<span class="token number">45</span>:<span class="token number">30.901234</span> <span class="token operator">|</span><span class="token operator">+</span><span class="token comment" spellcheck="true">-------------+----------------------------------+----------------------------+</span></code></pre><p>So, during application startup, we can do the following:</p><ol><li>Get the list of all applied migrations</li><li>Check the migrations dir to see if there are new unapplied migrations</li><li>Apply these pending migrations one by one</li></ol><p>That’s pretty much it for the simple version! Let’s implement the above logic.</p><h2 id="Implementation"><a href="#Implementation" class="headerlink" title="Implementation"></a>Implementation</h2><p>We’ll use Python and <a href="https://magicstack.github.io/asyncpg/current/" target="_blank" rel="noopener">asyncpg</a> for this implementation.</p><p>Let’s start with creating the <code>schema_migrations</code> table if it doesn’t exist:</p><pre class=" language-python"><code class="language-python"><span class="token keyword">import</span> asyncpg<span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">create_table</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>    query <span class="token operator">=</span> <span class="token triple-quoted-string string">"""        CREATE TABLE IF NOT EXISTS schema_migrations (            version     INTEGER   PRIMARY KEY,            name        TEXT      NOT NULL,            migrated_at TIMESTAMP NOT NULL DEFAULT NOW()        );    """</span>    connection <span class="token operator">=</span> <span class="token keyword">await</span> asyncpg<span class="token punctuation">.</span>connect<span class="token punctuation">(</span>user<span class="token operator">=</span><span class="token string">'postgres'</span><span class="token punctuation">)</span>    <span class="token keyword">await</span> connection<span class="token punctuation">.</span>execute<span class="token punctuation">(</span>query<span class="token punctuation">)</span>    <span class="token keyword">await</span> connection<span class="token punctuation">.</span>close<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre><p>Next, we get all the pending migrations that are not applied yet. For this we need to get all the migration files, retrieve the applied versions from <code>schema_migrations</code> table, and then filter out the migration files that are already applied.</p><pre class=" language-python"><code class="language-python"><span class="token keyword">from</span> pathlib <span class="token keyword">import</span> Path<span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">get_pending_migrations</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>    <span class="token comment" spellcheck="true"># Get all migration files</span>    migrations <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>    <span class="token keyword">for</span> path <span class="token keyword">in</span> Path<span class="token punctuation">(</span><span class="token string">'./src/migrations'</span><span class="token punctuation">)</span><span class="token punctuation">.</span>iterdir<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>        <span class="token keyword">if</span> <span class="token operator">not</span> path<span class="token punctuation">.</span>is_file<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>            <span class="token keyword">continue</span>        migration <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span>        migration<span class="token punctuation">[</span><span class="token string">"name"</span><span class="token punctuation">]</span> <span class="token operator">=</span> path<span class="token punctuation">.</span>name        migration<span class="token punctuation">[</span><span class="token string">"content"</span><span class="token punctuation">]</span> <span class="token operator">=</span> path<span class="token punctuation">.</span>read_text<span class="token punctuation">(</span><span class="token punctuation">)</span>        migration<span class="token punctuation">[</span><span class="token string">"version"</span><span class="token punctuation">]</span> <span class="token operator">=</span> int<span class="token punctuation">(</span>path<span class="token punctuation">.</span>name<span class="token punctuation">.</span>split<span class="token punctuation">(</span><span class="token string">"_"</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span>        migrations<span class="token punctuation">.</span>append<span class="token punctuation">(</span>migration<span class="token punctuation">)</span>    <span class="token comment" spellcheck="true"># Get all applied versions</span>    query <span class="token operator">=</span> <span class="token string">"SELECT version from schema_migrations ORDER BY version ASC"</span>    connection <span class="token operator">=</span> <span class="token keyword">await</span> asyncpg<span class="token punctuation">.</span>connect<span class="token punctuation">(</span>user<span class="token operator">=</span><span class="token string">'postgres'</span><span class="token punctuation">)</span>    records <span class="token operator">=</span> <span class="token keyword">await</span> connection<span class="token punctuation">.</span>fetch<span class="token punctuation">(</span>query<span class="token punctuation">)</span>    applied_versions <span class="token operator">=</span> <span class="token punctuation">[</span>r<span class="token punctuation">[</span><span class="token string">"version"</span><span class="token punctuation">]</span> <span class="token keyword">for</span> r <span class="token keyword">in</span> records<span class="token punctuation">]</span>    <span class="token keyword">await</span> connection<span class="token punctuation">.</span>close<span class="token punctuation">(</span><span class="token punctuation">)</span>    <span class="token comment" spellcheck="true"># Filter out applied migrations</span>    migrations <span class="token operator">=</span> <span class="token punctuation">[</span>m <span class="token keyword">for</span> m <span class="token keyword">in</span> migrations <span class="token keyword">if</span> m<span class="token punctuation">[</span><span class="token string">"version"</span><span class="token punctuation">]</span> <span class="token operator">not</span> <span class="token keyword">in</span> applied_versions<span class="token punctuation">]</span>    <span class="token comment" spellcheck="true"># Sort migrations by version</span>    migrations <span class="token operator">=</span> sorted<span class="token punctuation">(</span>migrations<span class="token punctuation">,</span> key<span class="token operator">=</span><span class="token keyword">lambda</span> m<span class="token punctuation">:</span> m<span class="token punctuation">[</span><span class="token string">'version'</span><span class="token punctuation">]</span><span class="token punctuation">)</span>    <span class="token keyword">return</span> migrations</code></pre><p>Finally, we apply these pending migrations one by one inside transactions.</p><pre class=" language-python"><code class="language-python"><span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">apply_pending_migrations</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>    <span class="token keyword">await</span> create_table<span class="token punctuation">(</span><span class="token punctuation">)</span>    migrations <span class="token operator">=</span> <span class="token keyword">await</span> get_pending_migrations<span class="token punctuation">(</span><span class="token punctuation">)</span>    connection <span class="token operator">=</span> <span class="token keyword">await</span> asyncpg<span class="token punctuation">.</span>connect<span class="token punctuation">(</span>user<span class="token operator">=</span><span class="token string">'postgres'</span><span class="token punctuation">)</span>    <span class="token keyword">try</span><span class="token punctuation">:</span>        <span class="token keyword">async</span> <span class="token keyword">with</span> connection<span class="token punctuation">.</span>transaction<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>            <span class="token keyword">for</span> migration <span class="token keyword">in</span> migrations<span class="token punctuation">:</span>                <span class="token keyword">await</span> connection<span class="token punctuation">.</span>execute<span class="token punctuation">(</span>migration<span class="token punctuation">[</span><span class="token string">"content"</span><span class="token punctuation">]</span><span class="token punctuation">)</span>                <span class="token keyword">await</span> connection<span class="token punctuation">.</span>execute<span class="token punctuation">(</span><span class="token string">"INSERT INTO schema_migrations (version, name) VALUES ($1, $2)"</span><span class="token punctuation">,</span> migration<span class="token punctuation">[</span><span class="token string">"version"</span><span class="token punctuation">]</span><span class="token punctuation">,</span> migration<span class="token punctuation">[</span><span class="token string">"name"</span><span class="token punctuation">]</span><span class="token punctuation">)</span>    <span class="token keyword">finally</span><span class="token punctuation">:</span>        <span class="token keyword">await</span> connection<span class="token punctuation">.</span>close<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre><p>That’s it! This is a simple implementation of a database schema migration script. This does forward-only migration, and applies all the pending migrations. This can be updated to perform rollbacks and migration to a specific version if needed. I’ve also omitted validation and error handling logic for sake of simplicity. These can be added based on your needs. </p><h2 id="Handling-breaking-changes"><a href="#Handling-breaking-changes" class="headerlink" title="Handling breaking changes"></a>Handling breaking changes</h2><p>One common issue with migrations is when you perform a breaking schema change. You have updated the application code to accommodate the new schema change, but the code that’s currently running in production would break if you apply this migration. </p><p>There are two ways to fix this:</p><p><strong>Downtime:</strong> This approach is simplest and straightforward. If your users are accommodating, you can take down your application, run the schema migrations and then restart the application with new code.</p><p><strong>Expand-Contract:</strong> If you can’t have downtime, then you need to use the “expand-contract” approach. Here you perform migrations multiple times. First you do a backwards compatible schema migration so that the application code running in production won’t break. The new code that is able to handle the new schema changes is deployed. We then perform another schema migration to remove the remnants of the old schema. This requires careful planning and can go wrong if it’s not tested properly before execution.</p><h2 id="Further-Reading"><a href="#Further-Reading" class="headerlink" title="Further Reading"></a>Further Reading</h2><ul><li><a href="https://docs.gitlab.com/ee/development/migration_style_guide.html" target="_blank" rel="noopener">GitLab Migration Style Guide</a></li><li><a href="https://postgres.ai/blog/20220525-common-db-schema-change-mistakes" target="_blank" rel="noopener">Common DB schema change mistakes</a></li><li><a href="https://xata.io/blog/migrations-and-exclusive-locks" target="_blank" rel="noopener">Schema changes and the Postgres lock queue</a></li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;Database schema migrations are rarely taught in universities, and if you’re a self-taught developer, migrations are not something you wou
      
    
    </summary>
    
    
      <category term="Python" scheme="https://www.sheshbabu.com/tags/Python/"/>
    
      <category term="Postgres" scheme="https://www.sheshbabu.com/tags/Postgres/"/>
    
      <category term="asyncpg" scheme="https://www.sheshbabu.com/tags/asyncpg/"/>
    
  </entry>
  
  <entry>
    <title>FastAPI without ORM: Getting started with asyncpg</title>
    <link href="https://www.sheshbabu.com/posts/fastapi-without-orm-getting-started-with-asyncpg/"/>
    <id>https://www.sheshbabu.com/posts/fastapi-without-orm-getting-started-with-asyncpg/</id>
    <published>2024-08-05T14:38:35.000Z</published>
    <updated>2024-08-05T15:51:40.796Z</updated>
    
    <content type="html"><![CDATA[<p>Let’s start with a simple FastAPI example:</p><pre class=" language-python"><code class="language-python"><span class="token comment" spellcheck="true"># src/main.py</span><span class="token keyword">import</span> uvicorn<span class="token keyword">from</span> fastapi <span class="token keyword">import</span> FastAPIapp <span class="token operator">=</span> FastAPI<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">"__main__"</span><span class="token punctuation">:</span>    uvicorn<span class="token punctuation">.</span>run<span class="token punctuation">(</span>app<span class="token punctuation">,</span> host<span class="token operator">=</span><span class="token string">"0.0.0.0"</span><span class="token punctuation">)</span></code></pre><p>Now, we want to add database logic to this app. We don’t want to use ORM, so we choose <a href="https://magicstack.github.io/asyncpg/current/" target="_blank" rel="noopener">asyncpg</a>. Let’s see how to integrate asyncpg with the above app.</p><p>We first create a <code>postgres.py</code> module that encapsulates the database connection, pool, termination, and other logic:</p><pre class=" language-python"><code class="language-python"><span class="token comment" spellcheck="true"># src/commons/postgres.py</span><span class="token keyword">import</span> asyncpgDATABASE_URL <span class="token operator">=</span> <span class="token string">"postgresql://postgres@localhost/postgres"</span><span class="token keyword">class</span> <span class="token class-name">Postgres</span><span class="token punctuation">:</span>    <span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> database_url<span class="token punctuation">:</span> str<span class="token punctuation">)</span><span class="token punctuation">:</span>        self<span class="token punctuation">.</span>database_url <span class="token operator">=</span> database_url    <span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">connect</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>        self<span class="token punctuation">.</span>pool <span class="token operator">=</span> <span class="token keyword">await</span> asyncpg<span class="token punctuation">.</span>create_pool<span class="token punctuation">(</span>self<span class="token punctuation">.</span>database_url<span class="token punctuation">)</span>    <span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">disconnect</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>        self<span class="token punctuation">.</span>pool<span class="token punctuation">.</span>close<span class="token punctuation">(</span><span class="token punctuation">)</span>database <span class="token operator">=</span> Postgres<span class="token punctuation">(</span>DATABASE_URL<span class="token punctuation">)</span></code></pre><p>I’ve hardcoded the database connection string here, but you can update this to pull from environment variables instead. </p><p>You’ll also notice that I’ve instantiated the <code>Postgres</code> class as <code>database</code> variable. This is intentional, and this module-scoped variable can be imported into other modules without the need for dependency injection.</p><p>Let’s update <code>main.py</code> to connect to the database:</p><pre class=" language-diff"><code class="language-diff">  # src/main.py  import uvicorn  from fastapi import FastAPI<span class="token inserted">+ from contextlib import asynccontextmanager</span><span class="token inserted">+ from src.commons.postgres import database</span><span class="token inserted">+ @asynccontextmanager</span><span class="token inserted">+ async def lifespan(app: FastAPI):</span><span class="token inserted">+     await database.connect()</span><span class="token inserted">+     yield</span><span class="token inserted">+     await database.disconnect()</span><span class="token deleted">- app = FastAPI()</span><span class="token inserted">+ app = FastAPI(lifespan=lifespan)</span>  if __name__ == "__main__":      uvicorn.run(app, host="0.0.0.0")</code></pre><p>We’ve added a <a href="https://fastapi.tiangolo.com/advanced/events/" target="_blank" rel="noopener">lifespan</a> function that connects to the database during application startup and then terminates the connection during shutdown.</p><p>Now that the setup is complete, let’s create an example table:</p><pre class=" language-sql"><code class="language-sql"><span class="token keyword">CREATE</span> <span class="token keyword">TABLE</span> <span class="token keyword">IF</span> <span class="token operator">NOT</span> <span class="token keyword">EXISTS</span> users <span class="token punctuation">(</span>    name  <span class="token keyword">TEXT</span>  <span class="token operator">NOT</span> <span class="token boolean">NULL</span><span class="token punctuation">,</span>    email <span class="token keyword">TEXT</span>  <span class="token operator">NOT</span> <span class="token boolean">NULL</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>We then create an equivalent <code>pydantic</code> model:</p><pre class=" language-python"><code class="language-python"><span class="token comment" spellcheck="true"># src/users/users_schema.py</span><span class="token keyword">from</span> pydantic <span class="token keyword">import</span> BaseModel<span class="token keyword">class</span> <span class="token class-name">User</span><span class="token punctuation">(</span>BaseModel<span class="token punctuation">)</span><span class="token punctuation">:</span>    name<span class="token punctuation">:</span> str    email<span class="token punctuation">:</span> str</code></pre><p>Let’s proceed to write logic for interacting with the above <code>users</code> table:</p><pre class=" language-python"><code class="language-python"><span class="token comment" spellcheck="true"># src/users/users_model.py</span><span class="token keyword">from</span> typing <span class="token keyword">import</span> List<span class="token punctuation">,</span> Optional<span class="token keyword">from</span> src<span class="token punctuation">.</span>commons<span class="token punctuation">.</span>postgres <span class="token keyword">import</span> database<span class="token keyword">from</span> src<span class="token punctuation">.</span>users<span class="token punctuation">.</span>users_schema <span class="token keyword">import</span> User<span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">get_user_by_email</span><span class="token punctuation">(</span>email<span class="token punctuation">:</span> str<span class="token punctuation">)</span> <span class="token operator">-</span><span class="token operator">></span> Optional<span class="token punctuation">[</span>User<span class="token punctuation">]</span><span class="token punctuation">:</span>    query <span class="token operator">=</span> <span class="token string">"SELECT name, email FROM users WHERE email = $1"</span>    <span class="token keyword">async</span> <span class="token keyword">with</span> database<span class="token punctuation">.</span>pool<span class="token punctuation">.</span>acquire<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token keyword">as</span> connection<span class="token punctuation">:</span>        row <span class="token operator">=</span> <span class="token keyword">await</span> connection<span class="token punctuation">.</span>fetchrow<span class="token punctuation">(</span>query<span class="token punctuation">,</span> email<span class="token punctuation">)</span>        <span class="token keyword">if</span> row <span class="token keyword">is</span> <span class="token operator">not</span> None<span class="token punctuation">:</span>            user <span class="token operator">=</span> User<span class="token punctuation">(</span>name<span class="token operator">=</span>row<span class="token punctuation">[</span><span class="token string">"name"</span><span class="token punctuation">]</span><span class="token punctuation">,</span> email<span class="token operator">=</span>row<span class="token punctuation">[</span><span class="token string">"email"</span><span class="token punctuation">]</span><span class="token punctuation">)</span>             <span class="token keyword">return</span> user        <span class="token keyword">return</span> None<span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">get_all_users</span><span class="token punctuation">(</span>limit<span class="token punctuation">:</span> int<span class="token punctuation">,</span> offset<span class="token punctuation">:</span> int<span class="token punctuation">)</span> <span class="token operator">-</span><span class="token operator">></span> List<span class="token punctuation">[</span>User<span class="token punctuation">]</span><span class="token punctuation">:</span>    query <span class="token operator">=</span> <span class="token string">"SELECT name, email FROM users LIMIT $1 OFFSET $2"</span>    <span class="token keyword">async</span> <span class="token keyword">with</span> database<span class="token punctuation">.</span>pool<span class="token punctuation">.</span>acquire<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token keyword">as</span> connection<span class="token punctuation">:</span>        rows <span class="token operator">=</span> <span class="token keyword">await</span> connection<span class="token punctuation">.</span>fetch<span class="token punctuation">(</span>query<span class="token punctuation">,</span> limit<span class="token punctuation">,</span> offset<span class="token punctuation">)</span>        users <span class="token operator">=</span> <span class="token punctuation">[</span>User<span class="token punctuation">(</span>name<span class="token operator">=</span>record<span class="token punctuation">[</span><span class="token string">"name"</span><span class="token punctuation">]</span><span class="token punctuation">,</span> email<span class="token operator">=</span>record<span class="token punctuation">[</span><span class="token string">"email"</span><span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token keyword">for</span> record <span class="token keyword">in</span> rows<span class="token punctuation">]</span>        <span class="token keyword">return</span> users<span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">insert_user</span><span class="token punctuation">(</span>user<span class="token punctuation">:</span> User<span class="token punctuation">)</span><span class="token punctuation">:</span>    query <span class="token operator">=</span> <span class="token string">"INSERT INTO users (name, email) VALUES ($1, $2)"</span>    <span class="token keyword">async</span> <span class="token keyword">with</span> database<span class="token punctuation">.</span>pool<span class="token punctuation">.</span>acquire<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token keyword">as</span> connection<span class="token punctuation">:</span>        <span class="token keyword">await</span> connection<span class="token punctuation">.</span>execute<span class="token punctuation">(</span>query<span class="token punctuation">,</span> user<span class="token punctuation">.</span>name<span class="token punctuation">,</span> user<span class="token punctuation">.</span>email<span class="token punctuation">)</span><span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">bulk_insert_users</span><span class="token punctuation">(</span>users<span class="token punctuation">:</span> List<span class="token punctuation">[</span>User<span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">:</span>    query <span class="token operator">=</span> <span class="token string">"INSERT INTO users (name, email) VALUES ($1, $2)"</span>    user_tuples <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">(</span>user<span class="token punctuation">.</span>name<span class="token punctuation">,</span> user<span class="token punctuation">.</span>email<span class="token punctuation">)</span> <span class="token keyword">for</span> user <span class="token keyword">in</span> users<span class="token punctuation">]</span>    <span class="token keyword">async</span> <span class="token keyword">with</span> database<span class="token punctuation">.</span>pool<span class="token punctuation">.</span>acquire<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token keyword">as</span> connection<span class="token punctuation">:</span>        <span class="token keyword">await</span> connection<span class="token punctuation">.</span>executemany<span class="token punctuation">(</span>query<span class="token punctuation">,</span> user_tuples<span class="token punctuation">)</span></code></pre><p>Okay, that’s a lot of code, but each function should be self-explanatory. As mentioned before, we’ve used the module-scoped variable <code>database</code> from <code>postgres.py</code> module. </p><p>We first acquire the connection and then use the specialized methods within it:</p><ul><li><code>fetchrow</code> is for querying a single row.</li><li><code>fetch</code> is for multiple rows.</li><li><code>execute</code> is when you want to execute multiple queries without returning rows.</li><li><code>executemany</code> is same as above and also accepts a list of arguments.</li></ul><p>One thing to note is that we need to perform “type conversion” before and after talking to asyncpg. You can see that we’re converting the <code>User</code> object in <code>insert_user</code> and <code>bulk_insert_users</code> to primitives or list of tuples before inserting them. We’re also converting the return value of asyncpg, which is a <a href="https://magicstack.github.io/asyncpg/current/api/index.html#asyncpg.Record" target="_blank" rel="noopener">dict-like object</a> to <code>User</code> type.</p><p>We can tie this together with the rest of the application by creating a router:</p><pre class=" language-python"><code class="language-python"><span class="token comment" spellcheck="true"># src/users/users_route.py</span><span class="token keyword">from</span> typing <span class="token keyword">import</span> Optional<span class="token punctuation">,</span> List<span class="token keyword">from</span> fastapi <span class="token keyword">import</span> APIRouter<span class="token keyword">from</span> src<span class="token punctuation">.</span>users <span class="token keyword">import</span> users_model<span class="token keyword">from</span> src<span class="token punctuation">.</span>users<span class="token punctuation">.</span>users_schema <span class="token keyword">import</span> Userusers_router <span class="token operator">=</span> APIRouter<span class="token punctuation">(</span>prefix<span class="token operator">=</span><span class="token string">"/users"</span><span class="token punctuation">)</span>@users_router<span class="token punctuation">.</span>get<span class="token punctuation">(</span><span class="token string">"/"</span><span class="token punctuation">)</span><span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">get_all_users</span><span class="token punctuation">(</span>limit<span class="token punctuation">:</span> Optional<span class="token punctuation">[</span>int<span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token number">10</span><span class="token punctuation">,</span> offset<span class="token punctuation">:</span> Optional<span class="token punctuation">[</span>int<span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">:</span>    <span class="token keyword">return</span> <span class="token keyword">await</span> users_model<span class="token punctuation">.</span>get_all_users<span class="token punctuation">(</span>limit<span class="token punctuation">,</span> offset<span class="token punctuation">)</span>@users_router<span class="token punctuation">.</span>get<span class="token punctuation">(</span><span class="token string">"/{email}"</span><span class="token punctuation">)</span><span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">get_user_by_email</span><span class="token punctuation">(</span>email<span class="token punctuation">:</span> str<span class="token punctuation">)</span><span class="token punctuation">:</span>    <span class="token keyword">return</span> <span class="token keyword">await</span> users_model<span class="token punctuation">.</span>get_user_by_email<span class="token punctuation">(</span>email<span class="token punctuation">)</span>@users_router<span class="token punctuation">.</span>post<span class="token punctuation">(</span><span class="token string">"/"</span><span class="token punctuation">)</span><span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">insert_user</span><span class="token punctuation">(</span>user<span class="token punctuation">:</span> User<span class="token punctuation">)</span><span class="token punctuation">:</span>    <span class="token keyword">return</span> <span class="token keyword">await</span> users_model<span class="token punctuation">.</span>insert_user<span class="token punctuation">(</span>user<span class="token punctuation">)</span>@users_router<span class="token punctuation">.</span>post<span class="token punctuation">(</span><span class="token string">"/bulk"</span><span class="token punctuation">)</span><span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">bulk_insert_users</span><span class="token punctuation">(</span>users<span class="token punctuation">:</span> List<span class="token punctuation">[</span>User<span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">:</span>    <span class="token keyword">return</span> <span class="token keyword">await</span> users_model<span class="token punctuation">.</span>bulk_insert_users<span class="token punctuation">(</span>users<span class="token punctuation">)</span></code></pre><p>And then adding it to <code>main.py</code>:</p><pre class=" language-diff"><code class="language-diff">  # src/main.py  import uvicorn  from fastapi import FastAPI  from contextlib import asynccontextmanager  from src.commons.postgres import database<span class="token inserted">+ from src.users.users_route import users_router</span>  @asynccontextmanager  async def lifespan(app: FastAPI):      await database.connect()      yield      await database.disconnect()  app = FastAPI(lifespan=lifespan)<span class="token inserted">+ app.include_router(users_router)</span>  if __name__ == "__main__":      uvicorn.run(app=app, host="0.0.0.0")</code></pre><p>You can find the full codebase in this <a href="https://github.com/sheshbabu/fastapi-asyncpg-demo" target="_blank" rel="noopener">repo</a>.</p><p>Related posts:</p><ul><li><a href="https://www.sheshbabu.com/posts/fastapi-structured-json-logging/">Structured JSON Logging using FastAPI</a></li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;Let’s start with a simple FastAPI example:&lt;/p&gt;
&lt;pre class=&quot; language-python&quot;&gt;&lt;code class=&quot;language-python&quot;&gt;&lt;span class=&quot;token comment&quot; sp
      
    
    </summary>
    
    
      <category term="Python" scheme="https://www.sheshbabu.com/tags/Python/"/>
    
      <category term="Postgres" scheme="https://www.sheshbabu.com/tags/Postgres/"/>
    
      <category term="asyncpg" scheme="https://www.sheshbabu.com/tags/asyncpg/"/>
    
      <category term="FastAPI" scheme="https://www.sheshbabu.com/tags/FastAPI/"/>
    
  </entry>
  
  <entry>
    <title>How Big Things Get Done</title>
    <link href="https://www.sheshbabu.com/posts/how-big-things-get-done/"/>
    <id>https://www.sheshbabu.com/posts/how-big-things-get-done/</id>
    <published>2024-08-04T10:47:13.000Z</published>
    <updated>2024-08-04T10:49:57.605Z</updated>
    
    <content type="html"><![CDATA[<p>If you’ve been in the industry for more than a few years, you might have seen at least one big project that failed badly - maybe it went significantly over budget, had its deadline repeatedly pushed back, or it delivered such a buggy product that users went back to using Excel. </p><p>Usually when people talk about such projects in casual settings, they would attribute the failure to incompetent leadership. As you grow in your career, you will slowly get asked to lead bigger and bigger projects. How do you avoid being that “incompetent leader”? Actually, what does competent leadership in this context look like? What do they do differently that make projects successful?</p><p>In their book <a href="https://sites.prh.com/how-big-things-get-done-book" target="_blank" rel="noopener">How Big Things Get Done</a>, the authors have researched what differentiates successful projects from the failed ones. </p><p>As the saying goes, <a href="https://en.m.wikipedia.org/wiki/Anna_Karenina_principle" target="_blank" rel="noopener"><em>All happy families are alike; each unhappy family is unhappy in its own way</em></a>. Similarly, according to the book, all successful projects exhibit the following attributes:</p><ul><li>Understanding Motivation</li><li>Thorough Planning</li><li>Forecasting Reliably</li><li>Prototyping Iteratively</li><li>Valuing Experience</li><li>Building High-Performing Teams</li></ul><h2 id="Understanding-Motivation"><a href="#Understanding-Motivation" class="headerlink" title="Understanding Motivation"></a>Understanding Motivation</h2><p>Most of the time, users don’t know what they want. So it’s important to ask questions at the beginning of the project to understand why they need something built, their fears, and their aspirations. Since projects are just vehicles for achieving some outcomes, by understanding the project’s goals, we can come up with better solutions.</p><p>If you don’t have users yet, it’s better to start with a short summary of the product, why it’s valuable to the users, and then work backwards to make sure the product fits in with the initial summary.</p><h2 id="Thorough-Planning"><a href="#Thorough-Planning" class="headerlink" title="Thorough Planning"></a>Thorough Planning</h2><h3 id="Two-Systems-of-Decision-Making"><a href="#Two-Systems-of-Decision-Making" class="headerlink" title="Two Systems of Decision Making"></a>Two Systems of Decision Making</h3><p>In the book <a href="https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow" target="_blank" rel="noopener">Thinking, Fast and Slow</a>, Daniel Kahneman says that humans have two decision making modes - “System One” and “System Two”. System One is the default mode. It is when we make snap judgments based on emotions or intuition, and this is how everyone operates in real life. This works great when done by experts as they have years of experience to draw upon. System Two is about conscious reasoning, where people correct the decision made through System One or even override it.</p><h3 id="Two-Patterns-of-Project-Planning"><a href="#Two-Patterns-of-Project-Planning" class="headerlink" title="Two Patterns of Project Planning"></a>Two Patterns of Project Planning</h3><p>This concept can be applied to how projects are planned and used to categorize the projects into either “Think fast, act slow” or “Think slow, act fast” patterns. </p><p>In the first pattern, people don’t spend too much time on planning and rush into development. There is rapid progress initially and everyone is happy. But because of superficial planning, we fail to uncover the problems and come up with potential solutions. So when the project hits these problems, it comes to a standstill. The project devolves into a mess.</p><p>The second pattern is the opposite. By doing thorough planning, we can get a realistic understanding of the timeline, costs and potential problems. We can then come up with solutions for those problems, change the scope of the project or just cancel it altogether if it didn’t make sense.</p><h3 id="Why-Planning-is-Skipped"><a href="#Why-Planning-is-Skipped" class="headerlink" title="Why Planning is Skipped"></a>Why Planning is Skipped</h3><p>System One thinking is not the only reason people avoid thorough planning. Sometimes it’s skipped for malicious purposes. For example, some might lowball the initial estimates so the project gets approved. Others might rush the project so no one can stop it because of sunk-cost fallacy.</p><h2 id="Forecasting-Reliably"><a href="#Forecasting-Reliably" class="headerlink" title="Forecasting Reliably"></a>Forecasting Reliably</h2><p>Forecasting errors is another reason why big projects fail. </p><h3 id="Anchors-and-Adjustments"><a href="#Anchors-and-Adjustments" class="headerlink" title="Anchors and Adjustments"></a>Anchors and Adjustments</h3><p>People usually base their estimates on how long similar projects in the past took. If they haven’t done a project of the same scale, they would use the estimates for a smaller project. This is called an “anchor”. They would then “adjust” these estimates for the bigger project. This is a natural way of thinking, but anchors are tricky and using wrong anchors would yield wrong estimates.</p><h3 id="Reference-Class-Forecasting"><a href="#Reference-Class-Forecasting" class="headerlink" title="Reference-Class Forecasting"></a>Reference-Class Forecasting</h3><p>All projects are unique in their own way. Maybe it’s the team, maybe it’s the economic situation, or it’s some unique combination of such factors. However, projects are not 100% unique. There would still be large parts of the project that are comparable to other projects in the same class. This is called a “reference-class”. It makes sense to use these already completed projects that are in the same class as our project. We use them as anchor and adjust the estimates based on how much our project differs from the mean in the class. The downside of this approach is that obtaining the numbers for the reference-class might not always be possible.</p><h2 id="Prototyping-Iteratively"><a href="#Prototyping-Iteratively" class="headerlink" title="Prototyping Iteratively"></a>Prototyping Iteratively</h2><h3 id="Why-Experimentation-is-Necessary"><a href="#Why-Experimentation-is-Necessary" class="headerlink" title="Why Experimentation is Necessary"></a>Why Experimentation is Necessary</h3><p>Planning is usually associated with images of endless meetings, bureaucratic paperwork and approval flows. However, these things are not the only aspects of planning. Experimentation and iteration are also part of planning as they help us come up with a detailed and reliable plan. If the project we’re starting has some unknown or risky elements, it’s better to experiment with some prototypes or proof-of-concepts (POC), so we know that those risky elements are solvable.</p><h3 id="How-Iterative-Planning-Works"><a href="#How-Iterative-Planning-Works" class="headerlink" title="How Iterative Planning Works"></a>How Iterative Planning Works</h3><p>Iterative design is also considered planning. If we split the project phases into “planning” and “development”, we would often notice that “development” is the most expensive part of the project. Whether it’s constructing a building, or producing a movie, etc. Whereas “planning” is relatively cheap and safe. We can iterate as many times as we want planning the building construction in CAD or storyboarding movies. The design starts with low fidelity and as it progresses, it gets more and more high fidelity. </p><p>For example, each Pixar movie starts with an idea, develops into a 12 page outline, then to a 120 page draft, to a detailed storyboard, to internal movie screening, and then finally to the world. Each stage goes through multiple iterations, and the script gets substantially rewritten and a good portion of the work done in initial phases are discarded. </p><p>While iterating like this is expensive, it gives us a lot of advantages:</p><ul><li>It gives freedom to the team for experimentation, to keep what works and discard what doesn’t.</li><li>It ensures that every part of the project gets scrutinized and tested before the development starts.</li><li>It makes sure that the team truly understands the problem before starting work.</li><li>It is cheaper to throw away mid-planning than during mid-development. </li></ul><p>In the software world, releasing an MVP (Minimum Viable Product) is also considered planning. It’s the act of actively experimenting with the product and releasing it early to see if it is what customers want.</p><h2 id="Valuing-Experience"><a href="#Valuing-Experience" class="headerlink" title="Valuing Experience"></a>Valuing Experience</h2><p>People, processes, and technology can have “experience”. A person can gain experience by spending a lot of time perfecting their craft. A process, technology or tool can also “gain” experience or become “proven” by getting iteratively refined. Trusting experts or using proven tools feels like common sense, but this needs to be said out loud.</p><h3 id="Experienced-People"><a href="#Experienced-People" class="headerlink" title="Experienced People"></a>Experienced People</h3><p>Experts know more, though there are two types of knowledge - “explicit knowledge” and “tacit knowledge”. Explicit knowledge can be obtained from books, classrooms or other sources, whereas tacit knowledge is the skilled intuition that only people with long experience have. </p><p><em>Sidenote: Big projects have a lot of moving parts like multiple teams, stakeholders, agendas etc, so a leader with more experience doing such projects would be able to navigate all these intuitively and make the project a success. This tacit knowledge gained from experience is not fully “serializable” to books or howto manuals (explicit knowledge). Though a novice who obtained explicit knowledge from books would be still better than one who has zero knowledge.</em></p><p>Experienced people often get aggressively marginalized or dismissed because leaders don’t appreciate how well it helps with project planning and delivery.</p><h3 id="Proven-Technology"><a href="#Proven-Technology" class="headerlink" title="Proven Technology"></a>Proven Technology</h3><p>Technology or systems can be thought of as “frozen experience”. People choose newer technology because they assume newer is better or they value custom-made or bespoke technology as they’re “unique”. But these are just “inexperienced technologies”.</p><p><em>Sidenote: This is similar to the <a href="https://mcfunley.com/choose-boring-technology" target="_blank" rel="noopener">boring technology</a> movement we’ve been seeing lately. This is where developers choose mature components like Postgres, Server Side Rendering, Monoliths, etc instead of whatever the current hotness is.</em></p><h2 id="Building-High-Performing-Teams"><a href="#Building-High-Performing-Teams" class="headerlink" title="Building High-Performing Teams"></a>Building High-Performing Teams</h2><p>Planning is very important for a project’s success, but a capable team implementing the plan is equally important. Big projects by nature often require people from multiple teams or external vendors work together to deliver it. </p><h3 id="Conflict-Resolution"><a href="#Conflict-Resolution" class="headerlink" title="Conflict Resolution"></a>Conflict Resolution</h3><p>It’s very common for conflicts to arise when multiple companies or teams are working together to deliver a project. When there are conflicts, it’s important for the leadership to intervene quickly and defuse the situation so it doesn’t get worse. Even better if contracts are designed to reward positive outcomes and to make sure that it’s in the interest of the vendors to work together to achieve a common goal.</p><h3 id="Identity-and-Purpose"><a href="#Identity-and-Purpose" class="headerlink" title="Identity and Purpose"></a>Identity and Purpose</h3><p>It’s important for people from multiple companies to feel like they’re working as a single team for the project instead of being employees of respective companies. Equally important, it’s for people to understand the outcomes of the project and why their work matters. These need to be said repeatedly for people to feel a sense of belonging and purpose as a single team.</p><h3 id="Open-Communication"><a href="#Open-Communication" class="headerlink" title="Open Communication"></a>Open Communication</h3><p>When leaders listen to the team’s feedback and quickly make the changes, it creates psychological safety. It improves the team morale, increases improvements, and problems are identified and fixed quickly.</p><p>None of these are easy. We need to invest a lot of time, effort and money to develop the team dynamics, but it makes a huge difference in project success.</p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>As we can see, a lot of these ideas are either straightforward to implement or considered common sense. What sets successful projects apart is how well these are executed. Once you know these ideas, it’s hard to unsee them in failing projects.</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;If you’ve been in the industry for more than a few years, you might have seen at least one big project that failed badly - maybe it went 
      
    
    </summary>
    
    
      <category term="Development" scheme="https://www.sheshbabu.com/tags/Development/"/>
    
      <category term="Summary" scheme="https://www.sheshbabu.com/tags/Summary/"/>
    
  </entry>
  
  <entry>
    <title>Graph Retrieval using Postgres Recursive CTEs</title>
    <link href="https://www.sheshbabu.com/posts/graph-retrieval-using-postgres-recursive-ctes/"/>
    <id>https://www.sheshbabu.com/posts/graph-retrieval-using-postgres-recursive-ctes/</id>
    <published>2024-07-28T08:37:10.000Z</published>
    <updated>2024-07-28T10:05:05.281Z</updated>
    
    <content type="html"><![CDATA[<p>Did you know you can use Postgres as a graph database for certain usecases?</p><p>Let’s say you have a graph like this:</p><p><img src="/images/2024-graph-retrieval-using-postgres-recursive-ctes/2024-graph-retrieval-using-postgres-recursive-ctes-01.png" alt=""></p><p>We can build this graph in NetworkX:</p><pre class=" language-python"><code class="language-python"><span class="token keyword">import</span> networkx <span class="token keyword">as</span> nxG <span class="token operator">=</span> nx<span class="token punctuation">.</span>Graph<span class="token punctuation">(</span><span class="token punctuation">)</span>G<span class="token punctuation">.</span>add_edges_from<span class="token punctuation">(</span><span class="token punctuation">[</span>    <span class="token punctuation">(</span><span class="token string">"A"</span><span class="token punctuation">,</span> <span class="token string">"B"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"A"</span><span class="token punctuation">,</span> <span class="token string">"C"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"A"</span><span class="token punctuation">,</span> <span class="token string">"D"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"A"</span><span class="token punctuation">,</span> <span class="token string">"E"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"A"</span><span class="token punctuation">,</span> <span class="token string">"F"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"E"</span><span class="token punctuation">,</span> <span class="token string">"G"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"F"</span><span class="token punctuation">,</span> <span class="token string">"G"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"E"</span><span class="token punctuation">,</span> <span class="token string">"H"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"H"</span><span class="token punctuation">,</span> <span class="token string">"I"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"D"</span><span class="token punctuation">,</span> <span class="token string">"I"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"J"</span><span class="token punctuation">,</span> <span class="token string">"K"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"C"</span><span class="token punctuation">,</span> <span class="token string">"K"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"C"</span><span class="token punctuation">,</span> <span class="token string">"L"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"B"</span><span class="token punctuation">,</span> <span class="token string">"L"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"B"</span><span class="token punctuation">,</span> <span class="token string">"M"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"N"</span><span class="token punctuation">,</span> <span class="token string">"O"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"P"</span><span class="token punctuation">,</span> <span class="token string">"R"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"P"</span><span class="token punctuation">,</span> <span class="token string">"Q"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"P"</span><span class="token punctuation">,</span> <span class="token string">"S"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"R"</span><span class="token punctuation">,</span> <span class="token string">"S"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">"Q"</span><span class="token punctuation">,</span> <span class="token string">"S"</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre><p>To store this in Postgres, create an <code>edges</code> table:</p><pre class=" language-sql"><code class="language-sql"><span class="token keyword">CREATE</span> <span class="token keyword">TABLE</span> <span class="token keyword">IF</span> <span class="token operator">NOT</span> <span class="token keyword">EXISTS</span> edges <span class="token punctuation">(</span>    u <span class="token keyword">TEXT</span><span class="token punctuation">,</span>    v <span class="token keyword">TEXT</span>    <span class="token comment" spellcheck="true">-- add other edge attributes</span><span class="token punctuation">)</span></code></pre><p>Insert the edges into the table:</p><pre class=" language-sql"><code class="language-sql"><span class="token keyword">INSERT</span> <span class="token keyword">INTO</span> edges <span class="token punctuation">(</span>u<span class="token punctuation">,</span> v<span class="token punctuation">)</span> <span class="token keyword">VALUES</span>    <span class="token punctuation">(</span><span class="token string">'A'</span><span class="token punctuation">,</span> <span class="token string">'B'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'A'</span><span class="token punctuation">,</span> <span class="token string">'C'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'A'</span><span class="token punctuation">,</span> <span class="token string">'D'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'A'</span><span class="token punctuation">,</span> <span class="token string">'E'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'A'</span><span class="token punctuation">,</span> <span class="token string">'F'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'B'</span><span class="token punctuation">,</span> <span class="token string">'L'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'B'</span><span class="token punctuation">,</span> <span class="token string">'M'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'C'</span><span class="token punctuation">,</span> <span class="token string">'K'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'C'</span><span class="token punctuation">,</span> <span class="token string">'L'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'D'</span><span class="token punctuation">,</span> <span class="token string">'I'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'E'</span><span class="token punctuation">,</span> <span class="token string">'G'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'E'</span><span class="token punctuation">,</span> <span class="token string">'H'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'F'</span><span class="token punctuation">,</span> <span class="token string">'G'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'H'</span><span class="token punctuation">,</span> <span class="token string">'I'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'J'</span><span class="token punctuation">,</span> <span class="token string">'K'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'N'</span><span class="token punctuation">,</span> <span class="token string">'O'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'P'</span><span class="token punctuation">,</span> <span class="token string">'R'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'P'</span><span class="token punctuation">,</span> <span class="token string">'Q'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'P'</span><span class="token punctuation">,</span> <span class="token string">'S'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'Q'</span><span class="token punctuation">,</span> <span class="token string">'S'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">(</span><span class="token string">'R'</span><span class="token punctuation">,</span> <span class="token string">'S'</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>Now that we have the graph stored, let’s see how to retrieve a <a href="https://en.wikipedia.org/wiki/Component_(graph_theory)" target="_blank" rel="noopener">connected component</a> from Postgres. If we want to retrieve all the nodes that are connected to “A”, we can use this recursive CTE:</p><pre class=" language-sql"><code class="language-sql"><span class="token keyword">WITH</span> RECURSIVE <span class="token number">cc</span> <span class="token keyword">AS</span> <span class="token punctuation">(</span>    <span class="token keyword">SELECT</span>        u<span class="token punctuation">,</span> v    <span class="token keyword">FROM</span>        edges    <span class="token keyword">WHERE</span>        u <span class="token operator">=</span> <span class="token string">'A'</span> <span class="token operator">OR</span> v <span class="token operator">=</span> <span class="token string">'A'</span>    <span class="token keyword">UNION</span>    <span class="token keyword">SELECT</span>        <span class="token number">e</span><span class="token punctuation">.</span>u<span class="token punctuation">,</span> <span class="token number">e</span><span class="token punctuation">.</span>v    <span class="token keyword">FROM</span>        edges <span class="token number">e</span>    <span class="token keyword">INNER</span> <span class="token keyword">JOIN</span>        <span class="token number">cc</span> <span class="token number">c</span> <span class="token keyword">ON</span>             <span class="token number">c</span><span class="token punctuation">.</span>u <span class="token operator">=</span> <span class="token number">e</span><span class="token punctuation">.</span>v <span class="token operator">OR</span>            <span class="token number">c</span><span class="token punctuation">.</span>v <span class="token operator">=</span> <span class="token number">e</span><span class="token punctuation">.</span>u <span class="token operator">OR</span>            <span class="token number">c</span><span class="token punctuation">.</span>v <span class="token operator">=</span> <span class="token number">e</span><span class="token punctuation">.</span>v <span class="token operator">OR</span>            <span class="token number">c</span><span class="token punctuation">.</span>u <span class="token operator">=</span> <span class="token number">e</span><span class="token punctuation">.</span>u<span class="token punctuation">)</span><span class="token keyword">SELECT</span> <span class="token operator">*</span> <span class="token keyword">FROM</span> <span class="token number">cc</span><span class="token punctuation">;</span></code></pre><p>Here’s how it works visually:</p><p><img src="/images/2024-graph-retrieval-using-postgres-recursive-ctes/2024-graph-retrieval-using-postgres-recursive-ctes-02.gif" alt=""></p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>I’m honestly surprised how well this works. I’ve tried this with graphs with millions of edges, and it works great! </p><p>The beauty of using Postgres to store graphs is that if your entities (nodes) are already stored inside other tables, then you’re able to keep both the entity metadata and relationships (graph) in the same database, thereby avoiding data synchronization between Postgres and a graph database.</p><p>Couple of caveats:</p><ul><li>The graphs I used are disjoint undirected graphs with millions of connected components. These are very common for <a href="https://www.sheshbabu.com/tags/Entity-Resolution/">entity resolution</a> use cases.</li><li>After retrieval, I use application-level libraries like NetworkX, igraph, etc to apply graph algorithms. This setup provides me with more graph algorithm options compared to the few available in graph databases like Neo4j etc.</li><li>If your graph is large or can have a <a href="https://mathworld.wolfram.com/GraphDiameter.html" target="_blank" rel="noopener">high diameter</a>, it’s better to add statement timeouts or limit the recursion depth so the above query doesn’t take a long time. Here’s an <a href="https://stackoverflow.com/a/51295400" target="_blank" rel="noopener">example</a> of how to implement them.</li><li>I’ve omitted things like constraints and indexes, these can be added based on your use case.</li></ul>]]></content>
    
    <summary type="html">
    
      Using Postgres as an alternative to graph databases for storing and retrieving graphs
    
    </summary>
    
    
      <category term="Python" scheme="https://www.sheshbabu.com/tags/Python/"/>
    
      <category term="Entity Resolution" scheme="https://www.sheshbabu.com/tags/Entity-Resolution/"/>
    
      <category term="Graph" scheme="https://www.sheshbabu.com/tags/Graph/"/>
    
      <category term="NetworkX" scheme="https://www.sheshbabu.com/tags/NetworkX/"/>
    
      <category term="Postgres" scheme="https://www.sheshbabu.com/tags/Postgres/"/>
    
  </entry>
  
  <entry>
    <title>Remove PDF password protection using qpdf and AppleScript</title>
    <link href="https://www.sheshbabu.com/posts/remove-pdf-passwords-using-qpdf-and-applescript/"/>
    <id>https://www.sheshbabu.com/posts/remove-pdf-passwords-using-qpdf-and-applescript/</id>
    <published>2024-07-21T09:13:30.000Z</published>
    <updated>2024-07-21T09:59:04.252Z</updated>
    
    <content type="html"><![CDATA[<p>For removing password protection from PDFs, I prefer using <a href="https://github.com/qpdf/qpdf" target="_blank" rel="noopener">qpdf</a> instead of installing bloated applications. One downside is that it needs to be invoked from Terminal, which is okay but not super convenient. I found a way to use macOS Automator to call this tool from Finder. </p><p><img src="/images/2024-qpdf-and-applescript/2024-qpdf-and-applescript-01.png" alt=""></p><p>Install qpdf from <a href="https://formulae.brew.sh/formula/qpdf" target="_blank" rel="noopener">HomeBrew</a></p><pre class=" language-shell"><code class="language-shell">$ brew install qpdf</code></pre><p>Open the builtin macOS application called “Automator”. Select “New Document” in the file picker dialog and choose “Quick Action” as the document type.</p><p>Set <code>Workflow receives current</code> dropdown to “PDF files” and <code>in</code> “Finder”. Drag and drop <code>Run AppleScript</code> action into the left pane. </p><p>Add the following code and save this Quick Action as “Remove Password”:</p><pre class=" language-applescript"><code class="language-applescript"><span class="token keyword">on</span> run <span class="token punctuation">{</span>input<span class="token punctuation">,</span> parameters<span class="token punctuation">}</span>    <span class="token keyword">set</span> the_file_path <span class="token keyword">to</span> POSIX path <span class="token keyword">of</span> <span class="token punctuation">(</span>input <span class="token operator">as</span> <span class="token class builtin">text</span><span class="token punctuation">)</span>    <span class="token keyword">set</span> AppleScript's <span class="token class builtin">text</span> item delimiters <span class="token keyword">to</span> <span class="token string">"/"</span>    <span class="token keyword">set</span> file_name <span class="token keyword">to</span> <span class="token keyword">last</span> <span class="token class builtin">text</span> item <span class="token keyword">of</span> the_file_path    <span class="token keyword">set</span> AppleScript's <span class="token class builtin">text</span> item delimiters <span class="token keyword">to</span> <span class="token string">"."</span>    <span class="token keyword">set</span> file_name_without_extension <span class="token keyword">to</span> <span class="token keyword">first</span> <span class="token class builtin">text</span> item <span class="token keyword">of</span> file_name    <span class="token keyword">set</span> AppleScript's <span class="token class builtin">text</span> item delimiters <span class="token keyword">to</span> <span class="token string">""</span>    <span class="token keyword">set</span> the_password <span class="token keyword">to</span> display dialog <span class="token string">"Enter the password for the PDF file:"</span> default answer <span class="token string">""</span> <span class="token keyword">with</span> icon caution    <span class="token keyword">set</span> the_password <span class="token keyword">to</span> <span class="token class builtin">text</span> returned <span class="token keyword">of</span> the_password    <span class="token keyword">set</span> the_output_path <span class="token keyword">to</span> <span class="token string">"/Users/"</span> <span class="token operator">&amp;</span> <span class="token punctuation">(</span>do shell <span class="token class builtin">script</span> <span class="token string">"echo $USER"</span><span class="token punctuation">)</span> <span class="token operator">&amp;</span> <span class="token string">"/Downloads/"</span> <span class="token operator">&amp;</span> file_name_without_extension <span class="token operator">&amp;</span> <span class="token string">"_decrypted.pdf"</span>    do shell <span class="token class builtin">script</span> "<span class="token operator">/</span>opt<span class="token operator">/</span>homebrew<span class="token operator">/</span>bin<span class="token operator">/</span>qpdf <span class="token comment" spellcheck="true">--password=" &amp; quoted form of the_password &amp; " --decrypt " &amp; quoted form of the_file_path &amp; " " &amp; quoted form of the_output_path</span>    display dialog <span class="token string">"The decrypted PDF file is saved as: "</span> <span class="token operator">&amp;</span> the_output_path<span class="token keyword">end</span> run</code></pre><p>This will show up when you right-click a PDF file in Finder under Quick Actions. Once decrypted, it saves the decrypted file to the user’s <code>Downloads</code> folder. </p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;For removing password protection from PDFs, I prefer using &lt;a href=&quot;https://github.com/qpdf/qpdf&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;qpdf&lt;/a&gt;
      
    
    </summary>
    
    
      <category term="AppleScript" scheme="https://www.sheshbabu.com/tags/AppleScript/"/>
    
  </entry>
  
  <entry>
    <title>Killing long running queries in Postgres</title>
    <link href="https://www.sheshbabu.com/posts/killing-long-running-queries-in-postgres/"/>
    <id>https://www.sheshbabu.com/posts/killing-long-running-queries-in-postgres/</id>
    <published>2024-07-20T09:32:04.000Z</published>
    <updated>2024-07-20T09:53:41.102Z</updated>
    
    <content type="html"><![CDATA[<p>To find all the queries that are currently running:</p><pre class=" language-sql"><code class="language-sql"><span class="token keyword">SELECT</span>    pid<span class="token punctuation">,</span>    AGE<span class="token punctuation">(</span><span class="token function">NOW</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> query_start<span class="token punctuation">)</span><span class="token punctuation">,</span>    query<span class="token keyword">FROM</span>    pg_stat_activity<span class="token keyword">WHERE</span>    query_start <span class="token operator">IS</span> <span class="token operator">NOT</span> <span class="token boolean">NULL</span><span class="token keyword">ORDER</span> <span class="token keyword">BY</span>    age <span class="token keyword">DESC</span></code></pre><p>The above will list all the running queries with its <code>pid</code>, <code>age</code> and <code>query</code>, where <code>age</code> is how long the query has been running.</p><p>To cancel a specific query, pass its <code>pid</code> to <code>pg_cancel_backend</code>:</p><pre class=" language-sql"><code class="language-sql"><span class="token keyword">SELECT</span> pg_cancel_backend<span class="token punctuation">(</span>pid<span class="token punctuation">)</span></code></pre><p>For example, if <code>pid</code> is <code>29212</code>:</p><pre class=" language-sql"><code class="language-sql"><span class="token keyword">SELECT</span> pg_cancel_backend<span class="token punctuation">(</span><span class="token number">29212</span><span class="token punctuation">)</span></code></pre><p>Note that sometimes <code>pg_cancel_backend</code> doesn’t work. In such cases, you will need to wait for the query to finish.</p>]]></content>
    
    <summary type="html">
    
      Finding and terminating long running queries in Postgres
    
    </summary>
    
    
      <category term="Postgres" scheme="https://www.sheshbabu.com/tags/Postgres/"/>
    
  </entry>
  
  <entry>
    <title>Implementing React-like Composition using Go&#39;s html/template</title>
    <link href="https://www.sheshbabu.com/posts/react-like-composition-using-go-html-template/"/>
    <id>https://www.sheshbabu.com/posts/react-like-composition-using-go-html-template/</id>
    <published>2024-05-22T22:13:31.000Z</published>
    <updated>2024-05-24T14:44:10.760Z</updated>
    
    <content type="html"><![CDATA[<p>React popularized component based frontend development and newer frameworks like Flutter, SwiftUI etc have followed the same model. I feel this model provides really good ergonomics and it’s hard to go back to the older ways of building frontends. </p><p>The problem with React is that it comes with the modern JS baggage - everything feels unnecessarily complex and bloated, and I say that as someone who has worked with JS and frontend for the past 15 years!</p><p>It’s no wonder I feel drawn to Go with its “batteries-included” design and no-BS philosophy. </p><p>But is it possible to get React like ergonomics in Go? Can we have our cake and eat it too? The answer is yes, and we can achieve all this using just the standard library. </p><p>If you’re too excited to see how it looks like, here’s a <a href="https://moujidemo.fly.dev/" target="_blank" rel="noopener">demo app</a> built using this approach. </p><p><img src="/images/2024-react-like-composition-using-go-html-template/2024-react-like-composition-using-go-html-template.png" alt=""></p><h2 id="Framing-the-problem"><a href="#Framing-the-problem" class="headerlink" title="Framing the problem"></a>Framing the problem</h2><p>We want to be able to split the UI into smaller components like <code>Button</code>, <code>Dropdown</code>, <code>FormInput</code> etc, then we compose them into bigger components like <code>Navbar</code>, <code>LoginForm</code>, <code>NewUserForm</code> etc, and then compose them further into even bigger page-level components like <code>HomePage</code>, <code>LoginPage</code>, and so on. </p><p>At the same time, we want these components to be reusable across the codebase, be configurable so they can have different variants and have the higher level components abstract away the lower details. </p><p>Let’s use an example, here’s how we want the frontend to look like visually:</p><p><img src="/images/2024-react-like-composition-using-go-html-template/2024-react-like-composition-using-go-html-template-01.png" alt=""></p><p>How the component tree would look like:</p><p><img src="/images/2024-react-like-composition-using-go-html-template/2024-react-like-composition-using-go-html-template-02.png" alt=""></p><p>We can see that the two pages share most of the components with some variations. Components like <code>Navbar</code> should be able to abstract away its child components. </p><h2 id="Implementation-using-html-template"><a href="#Implementation-using-html-template" class="headerlink" title="Implementation using html/template"></a>Implementation using html/template</h2><p>Lets define a low-level component, <code>Button</code>:</p><pre class=" language-go"><code class="language-go"><span class="token comment" spellcheck="true">// components/button.html</span><span class="token punctuation">{</span><span class="token punctuation">{</span>define <span class="token string">"button"</span><span class="token punctuation">}</span><span class="token punctuation">}</span>    <span class="token operator">&lt;</span>button class<span class="token operator">=</span><span class="token string">"button-container"</span><span class="token operator">></span>        <span class="token operator">&lt;</span>a            class<span class="token operator">=</span><span class="token string">"button {{if .IsPrimary}}primary{{end}}"</span>             <span class="token punctuation">{</span><span class="token punctuation">{</span><span class="token keyword">if</span> eq <span class="token punctuation">.</span>IsSubmit <span class="token boolean">true</span><span class="token punctuation">}</span><span class="token punctuation">}</span>                <span class="token keyword">type</span><span class="token operator">=</span><span class="token string">"submit"</span>            <span class="token punctuation">{</span><span class="token punctuation">{</span><span class="token keyword">else</span><span class="token punctuation">}</span><span class="token punctuation">}</span>                href<span class="token operator">=</span><span class="token string">"{{.Link}}"</span>            <span class="token punctuation">{</span><span class="token punctuation">{</span>end<span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token operator">></span>            <span class="token punctuation">{</span><span class="token punctuation">{</span><span class="token punctuation">.</span>Text<span class="token punctuation">}</span><span class="token punctuation">}</span>        <span class="token operator">&lt;</span><span class="token operator">/</span>a<span class="token operator">></span>    <span class="token operator">&lt;</span><span class="token operator">/</span>button<span class="token operator">></span><span class="token punctuation">{</span><span class="token punctuation">{</span>end<span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><pre class=" language-go"><code class="language-go"><span class="token comment" spellcheck="true">// components/button.go</span><span class="token keyword">package</span> components<span class="token keyword">type</span> Button <span class="token keyword">struct</span> <span class="token punctuation">{</span>    Text      <span class="token builtin">string</span>    Link      <span class="token builtin">string</span>    IsSubmit  <span class="token builtin">bool</span>    IsPrimary <span class="token builtin">bool</span><span class="token punctuation">}</span></code></pre><p>Let’s define the <code>Navbar</code>:</p><pre class=" language-go"><code class="language-go"><span class="token comment" spellcheck="true">// components/navbar.html</span><span class="token punctuation">{</span><span class="token punctuation">{</span>define <span class="token string">"navbar"</span><span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token operator">&lt;</span>div class<span class="token operator">=</span><span class="token string">"navbar"</span><span class="token operator">></span>    <span class="token operator">&lt;</span>a class<span class="token operator">=</span><span class="token string">"logo"</span> href<span class="token operator">=</span><span class="token string">"/"</span><span class="token operator">></span>My App<span class="token operator">&lt;</span><span class="token operator">/</span>a<span class="token operator">></span>    <span class="token operator">&lt;</span>div class<span class="token operator">=</span><span class="token string">"actions"</span><span class="token operator">></span>        <span class="token punctuation">{</span><span class="token punctuation">{</span>template <span class="token string">"dropdown"</span> <span class="token punctuation">.</span>ProjectsDropdown<span class="token punctuation">}</span><span class="token punctuation">}</span>        <span class="token punctuation">{</span><span class="token punctuation">{</span>template <span class="token string">"button"</span> <span class="token punctuation">.</span>SettingsButton<span class="token punctuation">}</span><span class="token punctuation">}</span>    <span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span><span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span><span class="token punctuation">{</span><span class="token punctuation">{</span>end<span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><pre class=" language-go"><code class="language-go"><span class="token comment" spellcheck="true">// components/navbar.go</span><span class="token keyword">package</span> components<span class="token keyword">type</span> Navbar <span class="token keyword">struct</span> <span class="token punctuation">{</span>    ProjectsDropdown Dropdown    SettingsButton   Button<span class="token punctuation">}</span></code></pre><p>As you can see, we’ve used <code>define</code> to define the components and <code>template</code> to compose smaller components within larger ones.</p><p>Here’s how we would use the above inside <code>LoginPage</code>:</p><pre class=" language-go"><code class="language-go"><span class="token comment" spellcheck="true">// features/login/login.html</span><span class="token operator">&lt;</span><span class="token operator">!</span>DOCTYPE html<span class="token operator">></span><span class="token operator">&lt;</span>html lang<span class="token operator">=</span><span class="token string">"en"</span><span class="token operator">></span>    <span class="token punctuation">{</span><span class="token punctuation">{</span>template <span class="token string">"head"</span><span class="token punctuation">}</span><span class="token punctuation">}</span>    <span class="token operator">&lt;</span>body<span class="token operator">></span>        <span class="token punctuation">{</span><span class="token punctuation">{</span>template <span class="token string">"navbar"</span> <span class="token punctuation">.</span>Navbar<span class="token punctuation">}</span><span class="token punctuation">}</span>        <span class="token operator">&lt;</span>div class<span class="token operator">=</span><span class="token string">"section"</span><span class="token operator">></span>            <span class="token operator">&lt;</span>div class<span class="token operator">=</span><span class="token string">"title"</span><span class="token operator">></span>Login<span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span>            <span class="token operator">&lt;</span>form action<span class="token operator">=</span><span class="token string">"/login"</span> method<span class="token operator">=</span><span class="token string">"post"</span><span class="token operator">></span>                <span class="token punctuation">{</span><span class="token punctuation">{</span>template <span class="token string">"input"</span> <span class="token punctuation">.</span>EmailInput<span class="token punctuation">}</span><span class="token punctuation">}</span>                <span class="token punctuation">{</span><span class="token punctuation">{</span>template <span class="token string">"input"</span> <span class="token punctuation">.</span>PasswordInput<span class="token punctuation">}</span><span class="token punctuation">}</span>                <span class="token punctuation">{</span><span class="token punctuation">{</span>template <span class="token string">"button"</span> <span class="token punctuation">.</span>SubmitButton<span class="token punctuation">}</span><span class="token punctuation">}</span>            <span class="token operator">&lt;</span><span class="token operator">/</span>form<span class="token operator">></span>        <span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span>    <span class="token operator">&lt;</span><span class="token operator">/</span>body<span class="token operator">></span><span class="token operator">&lt;</span><span class="token operator">/</span>html<span class="token operator">></span></code></pre><pre class=" language-go"><code class="language-go"><span class="token comment" spellcheck="true">// features/login/login.go</span><span class="token comment" spellcheck="true">// ...</span><span class="token keyword">func</span> <span class="token function">renderLoginForm</span><span class="token punctuation">(</span>w http<span class="token punctuation">.</span>ResponseWriter<span class="token punctuation">,</span> email <span class="token builtin">string</span><span class="token punctuation">,</span> emailError <span class="token builtin">string</span><span class="token punctuation">,</span> passwordError <span class="token builtin">string</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">type</span> templateData <span class="token keyword">struct</span> <span class="token punctuation">{</span>        Navbar        components<span class="token punctuation">.</span>Navbar        EmailInput    components<span class="token punctuation">.</span>Input        PasswordInput components<span class="token punctuation">.</span>Input        SubmitButton  components<span class="token punctuation">.</span>Button    <span class="token punctuation">}</span>    tmplData <span class="token operator">:=</span> templateData<span class="token punctuation">{</span>        Navbar<span class="token punctuation">:</span> components<span class="token punctuation">.</span>Navbar<span class="token punctuation">{</span>            SettingsButton<span class="token punctuation">:</span> components<span class="token punctuation">.</span>Button<span class="token punctuation">{</span>                Text<span class="token punctuation">:</span>     <span class="token string">"Settings"</span><span class="token punctuation">,</span>                Link<span class="token punctuation">:</span>     <span class="token string">"/settings"</span><span class="token punctuation">,</span>                IsSubmit<span class="token punctuation">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span>            <span class="token punctuation">}</span><span class="token punctuation">,</span>            ProjectsDropdown<span class="token punctuation">:</span> <span class="token comment" spellcheck="true">// ...</span>        <span class="token punctuation">}</span><span class="token punctuation">,</span>        EmailInput<span class="token punctuation">:</span> components<span class="token punctuation">.</span>Input<span class="token punctuation">{</span>            ID<span class="token punctuation">:</span>          <span class="token string">"email"</span><span class="token punctuation">,</span>            Label<span class="token punctuation">:</span>       <span class="token string">"Email"</span><span class="token punctuation">,</span>            Type<span class="token punctuation">:</span>        <span class="token string">"email"</span><span class="token punctuation">,</span>            Placeholder<span class="token punctuation">:</span> <span class="token string">"Enter your email address"</span><span class="token punctuation">,</span>            Error<span class="token punctuation">:</span>       emailError<span class="token punctuation">,</span>            Value<span class="token punctuation">:</span>       email<span class="token punctuation">,</span>        <span class="token punctuation">}</span><span class="token punctuation">,</span>        PasswordInput<span class="token punctuation">:</span> components<span class="token punctuation">.</span>Input<span class="token punctuation">{</span>            ID<span class="token punctuation">:</span>          <span class="token string">"password"</span><span class="token punctuation">,</span>            Label<span class="token punctuation">:</span>       <span class="token string">"Password"</span><span class="token punctuation">,</span>            Type<span class="token punctuation">:</span>        <span class="token string">"password"</span><span class="token punctuation">,</span>            Placeholder<span class="token punctuation">:</span> <span class="token string">"Enter your password"</span><span class="token punctuation">,</span>            Error<span class="token punctuation">:</span>       passwordError<span class="token punctuation">,</span>        <span class="token punctuation">}</span><span class="token punctuation">,</span>        SubmitButton<span class="token punctuation">:</span> components<span class="token punctuation">.</span>Button<span class="token punctuation">{</span>            Text<span class="token punctuation">:</span>      <span class="token string">"Login"</span><span class="token punctuation">,</span>            IsSubmit<span class="token punctuation">:</span>  <span class="token boolean">true</span><span class="token punctuation">,</span>            IsPrimary<span class="token punctuation">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span>        <span class="token punctuation">}</span><span class="token punctuation">,</span>    <span class="token punctuation">}</span>    templates<span class="token punctuation">.</span><span class="token function">Render</span><span class="token punctuation">(</span>w<span class="token punctuation">,</span> <span class="token string">"login.html"</span><span class="token punctuation">,</span> tmplData<span class="token punctuation">)</span><span class="token punctuation">}</span><span class="token comment" spellcheck="true">// ...</span></code></pre><p>Ignore the <code>templates.Render</code> part for the time being, I’ll explain it later. </p><p>You can see that we prep the data needed for rendering as a nested struct <code>tmplData</code> and then pass it to the template. The template which is already composed of multiple components splits <code>tmplData</code> and passes them to the relevant components. </p><p>If you’ve worked with React before, you can start seeing similarities in the way things are organized.</p><p>Notice how each page is also the root component, so we’ve multiple roots. I initially used the “template slots” pattern where there’s a single root template and others like <code>MainContent</code>, <code>Navbar</code>, <code>Sidebar</code> etc can “slot” into the placeholders, but I found it confusing and hard to reason about. The drawback with multiple roots approach is you end up repeating the <code>DOCTYPE</code>, <code>html</code>, and <code>body</code> tags, but I feel it’s managable. YMMV. </p><h2 id="Abstracting-away-details"><a href="#Abstracting-away-details" class="headerlink" title="Abstracting away details"></a>Abstracting away details</h2><p>We can repeat the same across multiple pages, but if we look at the UI design mockups above, the <code>Navbar</code> component should show the <code>Button</code> and <code>Dropdown</code> only in certain pages. And even in the pages where they’re shown, we don’t want to keep repeating this piece of code:</p><pre class=" language-go"><code class="language-go"><span class="token comment" spellcheck="true">// features/login/login.go</span><span class="token comment" spellcheck="true">// ...</span>    tmplData <span class="token operator">:=</span> templateData<span class="token punctuation">{</span>        Navbar<span class="token punctuation">:</span> components<span class="token punctuation">.</span>Navbar<span class="token punctuation">{</span>            SettingsButton<span class="token punctuation">:</span> components<span class="token punctuation">.</span>Button<span class="token punctuation">{</span>                Text<span class="token punctuation">:</span>     <span class="token string">"Settings"</span><span class="token punctuation">,</span>                Link<span class="token punctuation">:</span>     <span class="token string">"/settings"</span><span class="token punctuation">,</span>                IsSubmit<span class="token punctuation">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span>            <span class="token punctuation">}</span><span class="token punctuation">,</span>            ProjectsDropdown<span class="token punctuation">:</span> <span class="token comment" spellcheck="true">// ...</span>        <span class="token punctuation">}</span><span class="token punctuation">,</span><span class="token comment" spellcheck="true">// ...</span></code></pre><p>This code and whether to show/hide the <code>Button</code> and <code>Dropdown</code>  is better abstracted away. We can do this by creating a constructor that takes in an arg to toggle the Navbar behavior:</p><pre class=" language-diff"><code class="language-diff"> // components/navbar.go package components type Navbar struct {<span class="token inserted">+   ShouldShowActions bool</span>    ProjectsDropdown  Dropdown    SettingsButton    Button }<span class="token inserted">+func NewNavbar(ShouldShowActions bool) Navbar {</span><span class="token inserted">+    return Navbar{</span><span class="token inserted">+        ShouldShowActions: ShouldShowActions,</span><span class="token inserted">+        SettingsButton:    Button{Text: "Settings", Icon: "gear", Link: "/settings", IsSubmit: false},</span><span class="token inserted">+    }</span><span class="token inserted">+}</span></code></pre><pre class=" language-diff"><code class="language-diff"> // components/navbar.html {{define "navbar"}} &lt;div class="navbar">     &lt;a class="logo" href="/">My App&lt;/a><span class="token inserted">+   {{if .ShouldShowActions}}</span>     &lt;div class="actions">         {{template "dropdown" .ProjectsDropdown}}         {{template "button" .SettingsButton}}     &lt;/div><span class="token inserted">+   {{end}}</span> &lt;/div> {{end}}</code></pre><p>We can now update <code>LoginPage</code>:</p><pre class=" language-diff"><code class="language-diff"> // features/login/login.go // ...     tmplData := templateData{<span class="token deleted">-        Navbar: components.Navbar{</span><span class="token deleted">-            SettingsButton: components.Button{</span><span class="token deleted">-                Text:     "Settings",</span><span class="token deleted">-                Link:     "/settings",</span><span class="token deleted">-                IsSubmit: false,</span><span class="token deleted">-            },</span><span class="token deleted">-            ProjectsDropdown: // ...</span><span class="token deleted">-        },</span><span class="token inserted">+        Navbar: components.NewNavbar(false),</span> // ...</code></pre><h2 id="State-management"><a href="#State-management" class="headerlink" title="State management"></a>State management</h2><p>So far, we’ve seen how to compose components and pass data from one layer to the ones below. This is sufficient for simple applications, but if you’re building something like a dashboard, then we require state management.</p><p>Let’s say your dashboard has a project selector and a table with pagination, we would have these requirements:</p><ul><li>If no project is selected, we need to set default values</li><li>If the user clicks to see the next set of rows in a table, the project selector should not change</li><li>If project is changed, the table state should be reset</li></ul><p><img src="/images/2024-react-like-composition-using-go-html-template/2024-react-like-composition-using-go-html-template-03.png" alt=""></p><p>Basically interacting with some components should only affect itself, whereas interacting with others can affect multiple components. </p><p>We can use the OG solution for state management - URL query params. Here’s how the logic would look like: </p><pre class=" language-go"><code class="language-go"><span class="token comment" spellcheck="true">// ...</span><span class="token keyword">func</span> <span class="token function">HandleDashboardPage</span><span class="token punctuation">(</span>w http<span class="token punctuation">.</span>ResponseWriter<span class="token punctuation">,</span> r <span class="token operator">*</span>http<span class="token punctuation">.</span>Request<span class="token punctuation">)</span> <span class="token punctuation">{</span>    selectedProjectID <span class="token operator">:=</span> r<span class="token punctuation">.</span>URL<span class="token punctuation">.</span><span class="token function">Query</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">Get</span><span class="token punctuation">(</span><span class="token string">"project_id"</span><span class="token punctuation">)</span>    tableOffset <span class="token operator">:=</span> r<span class="token punctuation">.</span>URL<span class="token punctuation">.</span><span class="token function">Query</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">Get</span><span class="token punctuation">(</span><span class="token string">"table_offset"</span><span class="token punctuation">)</span>    <span class="token comment" spellcheck="true">// Set to default values</span>    <span class="token keyword">if</span> state<span class="token punctuation">.</span>selectedProjectID <span class="token operator">==</span> <span class="token string">""</span> <span class="token punctuation">{</span>        newURL <span class="token operator">:=</span> fmt<span class="token punctuation">.</span><span class="token function">Sprintf</span><span class="token punctuation">(</span><span class="token string">"/dashboard?project_id=%s&amp;table_offset=%d"</span><span class="token punctuation">,</span> projects<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>ProjectID<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span>        http<span class="token punctuation">.</span><span class="token function">Redirect</span><span class="token punctuation">(</span>w<span class="token punctuation">,</span> r<span class="token punctuation">,</span> newURL<span class="token punctuation">,</span> http<span class="token punctuation">.</span>StatusSeeOther<span class="token punctuation">)</span>        <span class="token keyword">return</span>    <span class="token punctuation">}</span>    <span class="token function">renderDashboardPage</span><span class="token punctuation">(</span>w<span class="token punctuation">,</span> selectedProjectID<span class="token punctuation">,</span> tableOffset<span class="token punctuation">)</span><span class="token punctuation">}</span><span class="token comment" spellcheck="true">// ...</span></code></pre><p>We parse the query params to see if the state is already set, otherwise we initialize it to default values. We then construct the links inside the table pagination to point to the new state.</p><pre class=" language-go"><code class="language-go"><span class="token comment" spellcheck="true">// ...</span><span class="token keyword">func</span> <span class="token function">renderDashboardPage</span><span class="token punctuation">(</span>w http<span class="token punctuation">.</span>ResponseWriter<span class="token punctuation">,</span> projectID <span class="token builtin">string</span><span class="token punctuation">,</span> tableOffset <span class="token builtin">string</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">type</span> templateData <span class="token keyword">struct</span> <span class="token punctuation">{</span>        Navbar components<span class="token punctuation">.</span>Navbar        Table  components<span class="token punctuation">.</span>Table    <span class="token punctuation">}</span>    navbar <span class="token operator">:=</span> <span class="token function">getNavbar</span><span class="token punctuation">(</span>projectID<span class="token punctuation">)</span>    table <span class="token operator">:=</span> <span class="token function">getTable</span><span class="token punctuation">(</span>projectID<span class="token punctuation">,</span> tableOffset<span class="token punctuation">)</span>    tmplData <span class="token operator">:=</span> templateData<span class="token punctuation">{</span>        Navbar<span class="token punctuation">:</span> navbar<span class="token punctuation">,</span>        Table<span class="token punctuation">:</span>  table<span class="token punctuation">,</span>    <span class="token punctuation">}</span>    templates<span class="token punctuation">.</span><span class="token function">Render</span><span class="token punctuation">(</span>w<span class="token punctuation">,</span> <span class="token string">"dashboard.html"</span><span class="token punctuation">,</span> tmplData<span class="token punctuation">)</span><span class="token punctuation">}</span><span class="token keyword">func</span> <span class="token function">getTable</span><span class="token punctuation">(</span>projectID <span class="token builtin">string</span><span class="token punctuation">,</span> offset <span class="token builtin">string</span><span class="token punctuation">)</span> components<span class="token punctuation">.</span>Table <span class="token punctuation">{</span>    <span class="token keyword">var</span> records <span class="token punctuation">[</span><span class="token punctuation">]</span>Record    limit <span class="token operator">:=</span> <span class="token number">10</span>    offset<span class="token punctuation">,</span> err <span class="token operator">:=</span> strconv<span class="token punctuation">.</span><span class="token function">Atoi</span><span class="token punctuation">(</span>offset<span class="token punctuation">)</span>    <span class="token keyword">if</span> err <span class="token operator">!=</span> <span class="token boolean">nil</span> <span class="token punctuation">{</span>        offset <span class="token operator">=</span> <span class="token number">0</span>    <span class="token punctuation">}</span>    table <span class="token operator">:=</span> components<span class="token punctuation">.</span>Table<span class="token punctuation">{</span>        Records<span class="token punctuation">:</span>              records<span class="token punctuation">,</span>        ShouldShowPagination<span class="token punctuation">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span>        Pagination<span class="token punctuation">:</span> components<span class="token punctuation">.</span>Pagination<span class="token punctuation">{</span>            PageStartRecord<span class="token punctuation">:</span> offset <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">,</span>            PageEndRecord<span class="token punctuation">:</span>   <span class="token number">0</span><span class="token punctuation">,</span>            TotalRecords<span class="token punctuation">:</span>    <span class="token number">0</span><span class="token punctuation">,</span>            PrevLink<span class="token punctuation">:</span>        <span class="token string">""</span><span class="token punctuation">,</span>            NextLink<span class="token punctuation">:</span>        <span class="token string">""</span><span class="token punctuation">,</span>        <span class="token punctuation">}</span><span class="token punctuation">,</span>    <span class="token punctuation">}</span>    records <span class="token operator">:=</span> <span class="token function">GetPaginatedRecords</span><span class="token punctuation">(</span>projectID<span class="token punctuation">,</span> limit<span class="token punctuation">,</span> offset<span class="token punctuation">)</span>    <span class="token keyword">if</span> <span class="token function">len</span><span class="token punctuation">(</span>records<span class="token punctuation">)</span> <span class="token operator">></span> <span class="token number">0</span> <span class="token punctuation">{</span>        table<span class="token punctuation">.</span>Records <span class="token operator">=</span> records        table<span class="token punctuation">.</span>Pagination<span class="token punctuation">.</span>TotalRecords <span class="token operator">=</span> records<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>TotalRecords        table<span class="token punctuation">.</span>Pagination<span class="token punctuation">.</span>PageStartRecord <span class="token operator">=</span> offset <span class="token operator">+</span> <span class="token number">1</span>        table<span class="token punctuation">.</span>Pagination<span class="token punctuation">.</span>PageEndRecord <span class="token operator">=</span> offset <span class="token operator">+</span> <span class="token function">len</span><span class="token punctuation">(</span>records<span class="token punctuation">)</span>        table<span class="token punctuation">.</span>ShouldShowPagination <span class="token operator">=</span> records<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>TotalRecords <span class="token operator">></span> limit    <span class="token punctuation">}</span>    <span class="token comment" spellcheck="true">// This would keep the projectID as constant and only change the table state</span>    <span class="token keyword">if</span> table<span class="token punctuation">.</span>ShouldShowPagination <span class="token operator">&amp;&amp;</span> offset <span class="token operator">!=</span> <span class="token number">0</span> <span class="token punctuation">{</span>        table<span class="token punctuation">.</span>Pagination<span class="token punctuation">.</span>PrevLink <span class="token operator">=</span> fmt<span class="token punctuation">.</span><span class="token function">Sprintf</span><span class="token punctuation">(</span><span class="token string">"/dashboard?project_id=%s&amp;table_offset=%d"</span><span class="token punctuation">,</span> projectID<span class="token punctuation">,</span> offset<span class="token operator">-</span>limit<span class="token punctuation">)</span>    <span class="token punctuation">}</span>    <span class="token keyword">if</span> table<span class="token punctuation">.</span>ShouldShowPagination <span class="token operator">&amp;&amp;</span> offset<span class="token operator">+</span>limit <span class="token operator">&lt;</span> table<span class="token punctuation">.</span>Pagination<span class="token punctuation">.</span>TotalRecords <span class="token punctuation">{</span>        table<span class="token punctuation">.</span>Pagination<span class="token punctuation">.</span>NextLink <span class="token operator">=</span> fmt<span class="token punctuation">.</span><span class="token function">Sprintf</span><span class="token punctuation">(</span><span class="token string">"/dashboard?project_id=%s&amp;table_offset=%d"</span><span class="token punctuation">,</span> projectID<span class="token punctuation">,</span> offset<span class="token operator">+</span>limit<span class="token punctuation">)</span>    <span class="token punctuation">}</span>    <span class="token keyword">return</span> table<span class="token punctuation">}</span><span class="token comment" spellcheck="true">// ...</span></code></pre><p>There’s probably a more elegant way to do this, but this approach is sufficient for my current use case. </p><h2 id="File-organization"><a href="#File-organization" class="headerlink" title="File organization"></a>File organization</h2><p>I wanted to colocate my templates with their logic instead of dumping them all into a single <code>/templates</code> folder:</p><pre class=" language-shell"><code class="language-shell">.|-- assets|   `-- index.css|-- commons|   |-- components|   |   |-- button.go|   |   |-- button.html|   |   |-- input.go|   |   |-- input.html|   |   |-- navbar.go|   |   `-- navbar.html|   `-- templates|       `-- templates.go|-- features|   |-- home|   |   |-- home.go|   |   `-- home.html|   |-- login|   |   |-- login.go|   |   `-- login.html|   `-- users|       |-- user_detail.html|       `-- users.go|-- go.mod|-- go.sum`-- main.go</code></pre><p>I also wanted to embed all these templates and css into a single Go binary. </p><p>Here’s how I implemented this:</p><pre class=" language-go"><code class="language-go"><span class="token comment" spellcheck="true">// main.go</span><span class="token keyword">package</span> main<span class="token keyword">import</span> <span class="token punctuation">(</span>    <span class="token string">"embed"</span>    <span class="token string">"myapp/commons/templates"</span><span class="token punctuation">)</span><span class="token comment" spellcheck="true">//go:embed all:commons all:features</span><span class="token keyword">var</span> resources embed<span class="token punctuation">.</span>FS<span class="token comment" spellcheck="true">//go:embed assets/*.css</span><span class="token keyword">var</span> assets embed<span class="token punctuation">.</span>FS<span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token comment" spellcheck="true">// ..</span>    templates<span class="token punctuation">.</span><span class="token function">NewTemplates</span><span class="token punctuation">(</span>resources<span class="token punctuation">)</span>    <span class="token comment" spellcheck="true">// ..</span><span class="token punctuation">}</span></code></pre><pre class=" language-go"><code class="language-go"><span class="token comment" spellcheck="true">// commons/templates/templates.go</span><span class="token keyword">package</span> templates<span class="token keyword">import</span> <span class="token punctuation">(</span>    <span class="token string">"bytes"</span>    <span class="token string">"embed"</span>    <span class="token string">"fmt"</span>    <span class="token string">"html/template"</span>    <span class="token string">"io/fs"</span>    <span class="token string">"net/http"</span>    <span class="token string">"strings"</span><span class="token punctuation">)</span><span class="token keyword">var</span> tmpl <span class="token operator">*</span>template<span class="token punctuation">.</span>Template <span class="token operator">=</span> <span class="token boolean">nil</span><span class="token keyword">func</span> <span class="token function">NewTemplates</span><span class="token punctuation">(</span>resources embed<span class="token punctuation">.</span>FS<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">var</span> paths <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token builtin">string</span>    fs<span class="token punctuation">.</span><span class="token function">WalkDir</span><span class="token punctuation">(</span>resources<span class="token punctuation">,</span> <span class="token string">"."</span><span class="token punctuation">,</span> <span class="token keyword">func</span><span class="token punctuation">(</span>path <span class="token builtin">string</span><span class="token punctuation">,</span> d fs<span class="token punctuation">.</span>DirEntry<span class="token punctuation">,</span> err <span class="token builtin">error</span><span class="token punctuation">)</span> <span class="token builtin">error</span> <span class="token punctuation">{</span>        <span class="token keyword">if</span> strings<span class="token punctuation">.</span><span class="token function">Contains</span><span class="token punctuation">(</span>d<span class="token punctuation">.</span><span class="token function">Name</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">".html"</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>            paths <span class="token operator">=</span> <span class="token function">append</span><span class="token punctuation">(</span>paths<span class="token punctuation">,</span> path<span class="token punctuation">)</span>        <span class="token punctuation">}</span>        <span class="token keyword">return</span> <span class="token boolean">nil</span>    <span class="token punctuation">}</span><span class="token punctuation">)</span>    tmpl <span class="token operator">=</span> template<span class="token punctuation">.</span><span class="token function">Must</span><span class="token punctuation">(</span>template<span class="token punctuation">.</span><span class="token function">ParseFS</span><span class="token punctuation">(</span>resources<span class="token punctuation">,</span> paths<span class="token operator">...</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">}</span><span class="token keyword">func</span> <span class="token function">Render</span><span class="token punctuation">(</span>w http<span class="token punctuation">.</span>ResponseWriter<span class="token punctuation">,</span> name <span class="token builtin">string</span><span class="token punctuation">,</span> data <span class="token keyword">interface</span><span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">var</span> buffer bytes<span class="token punctuation">.</span>Buffer    err <span class="token operator">:=</span> tmpl<span class="token punctuation">.</span><span class="token function">ExecuteTemplate</span><span class="token punctuation">(</span><span class="token operator">&amp;</span>buffer<span class="token punctuation">,</span> name<span class="token punctuation">,</span> data<span class="token punctuation">)</span>    <span class="token keyword">if</span> err <span class="token operator">!=</span> <span class="token boolean">nil</span> <span class="token punctuation">{</span>        err <span class="token operator">=</span> fmt<span class="token punctuation">.</span><span class="token function">Errorf</span><span class="token punctuation">(</span><span class="token string">"error executing template: %w"</span><span class="token punctuation">,</span> err<span class="token punctuation">)</span>        http<span class="token punctuation">.</span><span class="token function">Error</span><span class="token punctuation">(</span>w<span class="token punctuation">,</span> err<span class="token punctuation">.</span><span class="token function">Error</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> http<span class="token punctuation">.</span>StatusInternalServerError<span class="token punctuation">)</span>        <span class="token keyword">return</span>    <span class="token punctuation">}</span>    w<span class="token punctuation">.</span><span class="token function">Header</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">Set</span><span class="token punctuation">(</span><span class="token string">"Content-Type"</span><span class="token punctuation">,</span> <span class="token string">"text/html; charset=UTF-8"</span><span class="token punctuation">)</span>    buffer<span class="token punctuation">.</span><span class="token function">WriteTo</span><span class="token punctuation">(</span>w<span class="token punctuation">)</span><span class="token punctuation">}</span></code></pre><p>The <code>NewTemplates</code> function walks through all the template (<code>*.html</code>) files inside the project and parses them. We also see the <code>Render</code> function from earlier which renders the template and handles the errors.</p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>That’s it! I feel this setup should be sufficient for small to medium-sized frontend codebases.</p><p>You can see a full implementation <a href="https://github.com/sheshbabu/mouji" target="_blank" rel="noopener">here</a> or look at the <a href="https://moujidemo.fly.dev/" target="_blank" rel="noopener">demo app</a> built using this approach.</p>]]></content>
    
    <summary type="html">
    
      Compose and render frontend components using Go&#39;s standard library
    
    </summary>
    
    
      <category term="React" scheme="https://www.sheshbabu.com/tags/React/"/>
    
      <category term="Go" scheme="https://www.sheshbabu.com/tags/Go/"/>
    
  </entry>
  
  <entry>
    <title>What Every Programmer Should Know About Load Testing</title>
    <link href="https://www.sheshbabu.com/posts/what-every-programmer-should-know-about-load-testing/"/>
    <id>https://www.sheshbabu.com/posts/what-every-programmer-should-know-about-load-testing/</id>
    <published>2024-03-24T06:13:54.000Z</published>
    <updated>2024-05-19T06:17:28.272Z</updated>
    
    <content type="html"><![CDATA[<p>Load testing is very important to prevent reputational or revenue loss. However, most developers struggle to write load tests. Even if they do load testing, they don’t know how to do it properly. So they confidently release the app to prod, and when it inevitably fails to support more than 7 concurrent users, they scratch their heads wondering why it wasn’t caught earlier.</p><p>I suspect the term “test” in “load test” is throwing developers off. Load tests in essence are about simulating production traffic and see how our application handles it. If it was named something like “production traffic simulation”, I feel people would be far more serious about it.</p><p><img src="/images/2024-what-every-programmer-should-know-about-load-testing/2024-what-every-programmer-should-know-about-load-testing.png" alt=""></p><h2 id="What’s-the-plan"><a href="#What’s-the-plan" class="headerlink" title="What’s the plan?"></a>What’s the plan?</h2><p>Often the very first thing developers do is to pick their favourite load testing tool. It’s almost always written in the same language they’re familiar with - if they’re a Python developer, then <a href="https://locust.io/" target="_blank" rel="noopener">Locust</a>, if they work with JVM languages, it’s usually <a href="https://docs.gatling.io/" target="_blank" rel="noopener">Gatling</a> or <a href="https://jmeter.apache.org/" target="_blank" rel="noopener">JMeter</a>, if JavaScript, then <a href="https://k6.io/" target="_blank" rel="noopener">k6</a> or <a href="https://www.artillery.io/" target="_blank" rel="noopener">Artillery</a>, and so on. Their very next step would be to hammer their endpoints with arbitrary number of concurrent users with no sense of pacing or think time. And this test would be done against an environment that doesn’t have the same resources (CPU, memory, scaling factor etc) as the final prod environment.</p><blockquote><p>Load tests in essence are about simulating production traffic and see how our application handles it</p></blockquote><p>If you pause here and think about what’s happening, you’ll notice a couple of flaws. If load testing is simulating production traffic from real users, then we can’t just bombard our endpoints with random number of requests, can we? If we want to observe how our services deployed to production is going to handle the load, then shouldn’t the environment we’re testing against mirror the production in terms of resource (CPU, memory, scaling factor, warm cache, number of records in database etc) configs? The closer you get to modelling your expected user traffic and the resources you’ll have provisioned, the more accurate your test results are going to be. </p><blockquote><p>The closer you get to modelling your expected user traffic and the resources you’ll have provisioned, the more accurate your test results are going to be.</p></blockquote><p>Once you realize this, things start falling into place.</p><h2 id="How-many-users"><a href="#How-many-users" class="headerlink" title="How many users?"></a>How many users?</h2><p>Sometimes you get lucky and you’ll know exactly how many people are going to use your application. For example, if you’re building a webapp for a conference, you can derive the max number of users from the max number of attendees, and then add some buffer on top.</p><p>Other times, you’ll have a sense based on past events. If you’re building an ecommerce application and targeting 11.11 or christmas sales, you’ll have an idea of traffic and usage patterns from past year. You’ll also have data on how well your company is doing and how much you’re spending on marketing compared to previous years. You can extrapolate your future traffic based on these numbers.</p><p>If you’re launching a new application or preparing for an event you’ve never done before, it’s useful to do a soft launch to a small number of internal users to get an idea of the usage patterns, how your infrastructure is handling this reduced load, and more importantly, getting a sense of the response latency your initial users are willing to put up with. You can also do a load test with a small number of users (5-10) if you’re not able to get internal users to use at the same time. This type of test is called “smoke test”. </p><p>Another approach is to gradually increase the traffic in your load test until you reaching a breaking point. You start with a small number like 100 users and measure how your system is performing. Is the latency good ennough? Are you getting errors or crashes due to heavy load? How do the different services inside your system cope? Since different services scale differently, it’s useful to keep an eye on this. You then gradually increase this from 100 users to 1000 users and measure the same. You keep doing this until you start seeing degraded performance or errors. You’ll see some services/components being the bottleneck in your system. You can either tune them if possible or scale the resources available to that service (vertical or horizontal scaling). Rinse and repeat. The beauty of this approach is that you learn about the bottlenecks and the solutions that can fix these bottlenecks. When you ultimately release to prod and if there’s an unexpected amount of traffic, you’ll know exactly what services to scale instead of panicking.</p><h2 id="What-scenarios"><a href="#What-scenarios" class="headerlink" title="What scenarios?"></a>What scenarios?</h2><p>Now that you’ve an idea of the number of (virtual) users, we need to come up with what they would be doing during the load test - pages they visit, actions performed, how long they linger in each page, etc.</p><p>Unless the application you’re load testing is a simple blog, you’d be aware of the different usage patterns of different types of users. For example, in an ecommerce app, anonymous users mainly search, look at deals and browse the products, compared to logged in users who might add items to cart, checkout or look at their past orders. You’ll need to look at your logs or analytics to find out more about the types of users and what actions they perform in your application.</p><p><img src="/images/2024-what-every-programmer-should-know-about-load-testing/2024-what-every-programmer-should-know-about-load-testing-01.png" alt=""></p><p>After the above exercise, you’ll divide your virtual users into different user personas and create different user journeys so that your load test would closely simulate the real world. Once you’ve the different mapped out most of the user journeys (scenarios), you can prioritize few critical scenarios such as login, payment, or checkout, etc to write load tests for.</p><h2 id="What-to-measure"><a href="#What-to-measure" class="headerlink" title="What to measure?"></a>What to measure?</h2><p>After coming up with a realistic test plan and a traffic model, we need to come up with pass/fail metrics. </p><ul><li>What is considered a good enough response time? Is it 300ms, 1s, 5s or more? This differs from application to application and even varies within an application.</li><li>How many user sessions can your application support?</li><li>What is throughput (requests per second) you’re aiming for?</li><li>How many errors are you getting?</li><li>How do the resources like CPU, Memory etc react to increased load?</li><li>Does the auto scaling configuration work the way you want?</li><li>If a component crashes during the load test, should that be considered a failure or is it a non-critical service that can be down during heavy load?</li></ul><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>To summarize, load testing needs a comprehensive view of your product. It’s something you need to discuss within your team and come up with a realistic plan and metrics to track. </p><p>Aimlessly load testing helps no one, it actually gives your team false confidence that your application can indeed handle the load but would fail in production. If your application is critical for your company’s revenue or is high profile that it affects its reputation, then doing a proper load testing is a worthwhile undertaking.</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;Load testing is very important to prevent reputational or revenue loss. However, most developers struggle to write load tests. Even if th
      
    
    </summary>
    
    
      <category term="Performance" scheme="https://www.sheshbabu.com/tags/Performance/"/>
    
      <category term="Load Testing" scheme="https://www.sheshbabu.com/tags/Load-Testing/"/>
    
  </entry>
  
  <entry>
    <title>Taurus: Writing JMeter Load Tests As Code</title>
    <link href="https://www.sheshbabu.com/posts/taurus-writing-jmeter-load-tests-as-code/"/>
    <id>https://www.sheshbabu.com/posts/taurus-writing-jmeter-load-tests-as-code/</id>
    <published>2024-03-23T00:11:01.000Z</published>
    <updated>2024-05-19T06:17:28.272Z</updated>
    
    <content type="html"><![CDATA[<p>JMeter is one of the <a href="https://en.wikipedia.org/wiki/Apache_JMeter#Releases" target="_blank" rel="noopener">oldest</a> and widely used tool for load testing applications. However it’s GUI based, and JMX the storage format it uses for test plans is not user friendly (<a href="https://jmeter.apache.org/demos/ForEachTest2.jmx" target="_blank" rel="noopener">example file</a>). This makes it hard to collaborate with other team members.</p><p><a href="https://gettaurus.org/" target="_blank" rel="noopener">Taurus</a> is an open source tool that provides a friendly abstraction over JMeter. It allows one to write the test plans in a lightweight YAML format which are easier to read compared to JMX.</p><p><img src="/images/2024-taurus-writing-jmeter-load-tests-as-code/2024-taurus-writing-jmeter-load-tests-as-code.png" alt=""></p><h2 id="Installation"><a href="#Installation" class="headerlink" title="Installation"></a>Installation</h2><p>Follow the installation steps from Taurus <a href="https://gettaurus.org/install/Installation/" target="_blank" rel="noopener">here</a></p><h2 id="Simple-test-plan"><a href="#Simple-test-plan" class="headerlink" title="Simple test plan"></a>Simple test plan</h2><p>Let’s come up with a very simple test plan to familiarize ourselves with Taurus.</p><pre class=" language-yaml"><code class="language-yaml"><span class="token key atrule">scenarios</span><span class="token punctuation">:</span>  <span class="token key atrule">home</span><span class="token punctuation">:</span>    <span class="token key atrule">requests</span><span class="token punctuation">:</span>      <span class="token punctuation">-</span> <span class="token key atrule">url</span><span class="token punctuation">:</span> http<span class="token punctuation">:</span>//localhost<span class="token punctuation">:</span>8000/home<span class="token key atrule">execution</span><span class="token punctuation">:</span>  <span class="token punctuation">-</span> <span class="token key atrule">scenario</span><span class="token punctuation">:</span> home    <span class="token key atrule">concurrency</span><span class="token punctuation">:</span> <span class="token number">1</span>    <span class="token key atrule">iterations</span><span class="token punctuation">:</span> <span class="token number">10</span></code></pre><p>This will make a HTTP GET request to the endpoint <code>http://localhost:8000/home</code> for 10 times.</p><p>Save the above as <code>main.yaml</code> and run using <code>bzt main.yaml</code>. For learning purposes, you can create a dummy server by running <code>python3 -m http.server</code> and create dummy “endpoints” using <code>touch home account</code></p><p>In real world applications, we’ll obviously make more than one request. We can specify sequential requests as follows:</p><pre class=" language-diff"><code class="language-diff">  scenarios:    home:      requests:        - url: http://localhost:8000/home<span class="token inserted">+       - url: http://localhost:8000/account</span>  execution:    - scenario: home      concurrency: 1      iterations: 10</code></pre><p>Let’s also add a POST request:</p><pre class=" language-diff"><code class="language-diff">  scenarios:    home:      requests:        - url: http://localhost:8000/home        - url: http://localhost:8000/account<span class="token inserted">+       - url: http://localhost:8000/login</span><span class="token inserted">+         method: POST</span><span class="token inserted">+         body: '{ username:"user", password:"hunter2" }'</span>  execution:    - scenario: home      concurrency: 1      iterations: 10</code></pre><p>Then add some headers and cookies:</p><pre class=" language-diff"><code class="language-diff">  scenarios:    home:<span class="token inserted">+     headers:</span><span class="token inserted">+       Cookie: 'connect.sid=abc'</span><span class="token inserted">+       authorization: Bearer 123</span>      requests:        - url: http://localhost:8000/home        - url: http://localhost:8000/account        - url: http://localhost:8000/login          method: POST          body: '{ username:"user", password:"hunter2" }'  execution:    - scenario: home      concurrency: 1      iterations: 10</code></pre><p>More examples <a href="https://gettaurus.org/docs/JMeter/#Requests" target="_blank" rel="noopener">here</a>.</p><h2 id="Parameterizing-values"><a href="#Parameterizing-values" class="headerlink" title="Parameterizing values"></a>Parameterizing values</h2><p>We can pass values from command line to the test plan using “JMeter properties”. Say we want to configure the <code>iterations</code> from command line, we can run the test using:</p><pre class=" language-shell"><code class="language-shell">bzt main.yaml -o modules.jmeter.properties.iterations=20</code></pre><p>and change our script to use this parameter:</p><pre class=" language-diff"><code class="language-diff">  scenarios:    home:      requests:        - url: http://localhost:8000/home        - url: http://localhost:8000/account  execution:    - scenario: home      concurrency: 1<span class="token deleted">-     iterations: 10</span><span class="token inserted">+     iterations: ${__P(iterations)}</span></code></pre><p>Multiple parameters can be passed this way. More examples and configuration options <a href="https://gettaurus.org/docs/JMeter/#JMeter-Properties-and-Variables" target="_blank" rel="noopener">here</a>.</p><h2 id="Splitting-scenarios-into-multiple-files"><a href="#Splitting-scenarios-into-multiple-files" class="headerlink" title="Splitting scenarios into multiple files"></a>Splitting scenarios into multiple files</h2><p>As your application becomes more complex, your scenarios would also become longer. At some point, you’d need to split your scenarios into multiple files. Let’s move the <code>home</code> scenario to a new file <code>home.yaml</code> and import it into <code>main.yaml</code></p><pre class=" language-diff"><code class="language-diff"><span class="token inserted">+ included-configs:</span><span class="token inserted">+   - home.yaml</span><span class="token deleted">- scenarios:</span><span class="token deleted">-   home:</span><span class="token deleted">-     requests:</span><span class="token deleted">-       - url: http://localhost:8000/home</span><span class="token deleted">-       - url: http://localhost:8000/account</span>  execution:    - scenario: home      concurrency: 1      iterations: 10</code></pre><pre class=" language-yaml"><code class="language-yaml"><span class="token comment" spellcheck="true"># home.yaml</span><span class="token key atrule">scenarios</span><span class="token punctuation">:</span>  <span class="token key atrule">home</span><span class="token punctuation">:</span>    <span class="token key atrule">requests</span><span class="token punctuation">:</span>      <span class="token punctuation">-</span> <span class="token key atrule">url</span><span class="token punctuation">:</span> http<span class="token punctuation">:</span>//localhost<span class="token punctuation">:</span>8000/home      <span class="token punctuation">-</span> <span class="token key atrule">url</span><span class="token punctuation">:</span> http<span class="token punctuation">:</span>//localhost<span class="token punctuation">:</span>8000/account</code></pre><h2 id="Using-data-from-CSV-file"><a href="#Using-data-from-CSV-file" class="headerlink" title="Using data from CSV file"></a>Using data from CSV file</h2><p>Sometimes you would need to test your endpoints with multiple inputs. The <code>data sources</code> feature comes in handy those times. You can save your inputs as a CSV file and refer them in test file.</p><pre class=" language-diff"><code class="language-diff">  # home.yaml  scenarios:    home:<span class="token inserted">+     data-sources:</span><span class="token inserted">+       - path: customers.csv</span><span class="token inserted">+         random-order: false</span>      requests:        - url: http://localhost:8000/home<span class="token deleted">-       - url: http://localhost:8000/account</span><span class="token inserted">+       - url: http://localhost:8000/account?customer_id=${customer_id}&amp;customer_name=${customer_name}</span></code></pre><p>Contents of <code>customers.csv</code>:</p><pre class=" language-csv"><code class="language-csv">customer_id,customer_name1,John Doe2,Jane Smith3,Michael Johnson</code></pre><p>As you can see above, the columns in the CSV file get converted to variables. These variables are initialized with values from one row of the file for each request and can be used in urls, headers etc. More options <a href="https://gettaurus.org/docs/DataSources/" target="_blank" rel="noopener">here</a>.</p><h2 id="Using-scripts-for-processing-requests"><a href="#Using-scripts-for-processing-requests" class="headerlink" title="Using scripts for processing requests"></a>Using scripts for processing requests</h2><p>Sometimes you need to pass the output of an endpoint into the input of a subsequent endpoint. This can’t be done using YAML, so we need to rely on scripting languages like JavaScript, Groovy etc. </p><pre class=" language-yaml"><code class="language-yaml"><span class="token comment" spellcheck="true"># home.yaml</span><span class="token key atrule">scenarios</span><span class="token punctuation">:</span>  <span class="token key atrule">home</span><span class="token punctuation">:</span>    <span class="token key atrule">requests</span><span class="token punctuation">:</span>      <span class="token punctuation">-</span> <span class="token key atrule">url</span><span class="token punctuation">:</span> http<span class="token punctuation">:</span>//localhost<span class="token punctuation">:</span>8000/home      <span class="token punctuation">-</span> <span class="token key atrule">url</span><span class="token punctuation">:</span> http<span class="token punctuation">:</span>//localhost<span class="token punctuation">:</span>8000/account      <span class="token punctuation">-</span> <span class="token key atrule">url</span><span class="token punctuation">:</span> http<span class="token punctuation">:</span>//localhost<span class="token punctuation">:</span>8000/user        <span class="token key atrule">jsr223</span><span class="token punctuation">:</span>          <span class="token punctuation">-</span> <span class="token key atrule">language</span><span class="token punctuation">:</span> javascript            <span class="token key atrule">execute</span><span class="token punctuation">:</span> after            <span class="token key atrule">script-text</span><span class="token punctuation">:</span> <span class="token punctuation">|</span><span class="token scalar string">              var response = JSON.parse(prev.getResponseDataAsString());              var userId = response.userId;              print(userId);              vars.put("user-id", userId);</span>      <span class="token punctuation">-</span> <span class="token key atrule">url</span><span class="token punctuation">:</span> http<span class="token punctuation">:</span>//localhost<span class="token punctuation">:</span>8000/log        <span class="token key atrule">method</span><span class="token punctuation">:</span> POST        <span class="token key atrule">headers</span><span class="token punctuation">:</span>          <span class="token key atrule">Content-Type</span><span class="token punctuation">:</span> application/json        <span class="token key atrule">jsr223</span><span class="token punctuation">:</span>          <span class="token punctuation">-</span> <span class="token key atrule">language</span><span class="token punctuation">:</span> javascript            <span class="token key atrule">execute</span><span class="token punctuation">:</span> before            <span class="token key atrule">script-text</span><span class="token punctuation">:</span> <span class="token punctuation">|</span><span class="token scalar string">              var requestBody = {                "userId": vars.get("user-id")              };              vars.put("request-body", JSON.stringify(requestBody));</span>        <span class="token key atrule">body</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>request<span class="token punctuation">-</span>body<span class="token punctuation">}</span></code></pre><p>In this example, we add a script that executes after the <code>/user</code> request to extract the <code>userId</code> and use it before the <code>/log</code> request to construct a payload.</p><p>Notice how I used <code>print()</code> instead of <code>console.log()</code> for debugging. This is because the <a href="https://www.oracle.com/technical-resources/articles/java/jf14-nashorn.html" target="_blank" rel="noopener">JavaScript used in JMeter</a> is different from the ones used in Web Browsers or NodeJS, so some keywords or APIs would not work. Also, JMeter developers recommend using <a href="https://jmeter.apache.org/usermanual/best-practices.html#jsr223" target="_blank" rel="noopener">Groovy instead of JavaScript</a> if you’re doing heavy load tests.</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;JMeter is one of the &lt;a href=&quot;https://en.wikipedia.org/wiki/Apache_JMeter#Releases&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;oldest&lt;/a&gt; and widely 
      
    
    </summary>
    
    
      <category term="Performance" scheme="https://www.sheshbabu.com/tags/Performance/"/>
    
      <category term="Load Testing" scheme="https://www.sheshbabu.com/tags/Load-Testing/"/>
    
  </entry>
  
  <entry>
    <title>Analyzing Slow Python Code using cProfile</title>
    <link href="https://www.sheshbabu.com/posts/analyzing-slow-python-code-using-cprofile/"/>
    <id>https://www.sheshbabu.com/posts/analyzing-slow-python-code-using-cprofile/</id>
    <published>2024-03-22T10:49:00.000Z</published>
    <updated>2024-05-19T06:17:28.270Z</updated>
    
    <content type="html"><![CDATA[<p>Python is widely considered to be a slow language. However it’s slow only when compared to other fast languages. I’d argue it’s a perfectly fine language for most applications - It’s very easy to learn which makes it ideal for teams with junior developers and its simplicity allows developers to focus on solving problems instead of worrying about low level details.</p><p>So it’s no surprise when lot of companies start with Python in the beginning. Most companies fail, so might as well fail faster. If they succeed and start getting more customers, they start hitting the performance limitations of Python. At this point, they can throw away the code and rewrite in a more performant language like Rust, but it’s better to profile your application and see where the bottlenecks are. Most of the time all you need to do is to find and fix these slow functions.</p><p><img src="/images/2024-analyzing-slow-python-code-using-cprofile/analyzing-slow-python-code-using-cprofile.png" alt=""></p><p>Before we jump into profiling, let’s build an intuition for flame graphs.<br>Let’s start with a simple example:</p><pre class=" language-python"><code class="language-python"><span class="token comment" spellcheck="true"># test.py</span><span class="token keyword">from</span> time <span class="token keyword">import</span> sleepsleep<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span></code></pre><p>This script will take 1 second to run and it can be visualized as:</p><p><img src="/images/2024-analyzing-slow-python-code-using-cprofile/analyzing-slow-python-code-using-cprofile-01.png" alt=""></p><p>The horizontal bar indicates the amount of time. Let’s add a function around it:</p><pre class=" language-python"><code class="language-python"><span class="token comment" spellcheck="true"># test.py</span><span class="token keyword">from</span> time <span class="token keyword">import</span> sleep<span class="token keyword">def</span> <span class="token function">a</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>    sleep<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span>a<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre><p>This would look like:</p><p><img src="/images/2024-analyzing-slow-python-code-using-cprofile/analyzing-slow-python-code-using-cprofile-02.png" alt=""></p><p>Notice how even though <code>a()</code> doesn’t do anything except call <code>sleep()</code>, it still has the same length. This is because these bars are cumulative, that is they add up. Which makes sense as the time spent on a function should be the total time of all the functions it calls and so on. In cProfile, this is called as <code>cumtime</code>. The time spent only on <code>a()</code> which is close to <code>0</code> is known as total time or <code>tottime</code>.</p><p>Let’s look at a one more example:</p><pre class=" language-python"><code class="language-python"><span class="token comment" spellcheck="true"># test.py</span><span class="token keyword">from</span> time <span class="token keyword">import</span> sleep<span class="token keyword">def</span> <span class="token function">a</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>    sleep<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span>    b<span class="token punctuation">(</span><span class="token punctuation">)</span>    sleep<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token keyword">def</span> <span class="token function">b</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>    sleep<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span>a<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre><p><img src="/images/2024-analyzing-slow-python-code-using-cprofile/analyzing-slow-python-code-using-cprofile-03.png" alt=""></p><p>Here we can see that even though call <code>sleep()</code> twice in <code>a()</code>, it’s been collapsed into a single bar. This is because in real programs, there’ll be thousands or millions of calls to a function which can’t all be represented separately.</p><p>Since <code>sleep()</code> is called 3 times, the <code>ntimes</code> value in cProfile would be <code>3</code>. We know the <code>tottime</code> and <code>cumtime</code>, and these sum up the time taken for all calls. To get a <code>percall</code> value for a function, we need to divide <code>tottime</code> and <code>cumtime</code> by <code>ntimes</code> . For example, here since <code>tottime</code> for <code>sleep()</code> is 3s, the <code>percall</code> value for <code>tottime</code> would be <code>1</code>.</p><p>This should be sufficient to understand the flame graphs. Note that in the above examples, I used a variation of flame graphs called icicle graphs which are same as flame graphs but upside down.</p><p>To profile the above code, we need to make changes as follows:</p><pre class=" language-diff"><code class="language-diff"># test.pyfrom time import sleep<span class="token inserted">+ import cProfile, pstats</span>def a():    sleep(1)    b()    sleep(1)def b():    sleep(1)<span class="token inserted">+ profiler = cProfile.Profile()</span><span class="token inserted">+ profiler.enable()</span>a()<span class="token inserted">+ profiler.disable()</span><span class="token inserted">+ stats = pstats.Stats(profiler)</span><span class="token inserted">+ stats.dump_stats("./cProfile.stats")</span><span class="token inserted">+ stats.print_stats()</span></code></pre><p>This can be run as <code>python3 test.py</code> and you’ll see the following results in the terminal:</p><pre class=" language-shell"><code class="language-shell">$ python3 test.py         6 function calls in 3.004 seconds   Random listing order was used   ncalls  tottime  percall  cumtime  percall filename:lineno(function)        3    3.004    1.001    3.004    1.001 {built-in method time.sleep}        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}        1    0.000    0.000    3.004    3.004 /home/user/test.py:5(a)        1    0.000    0.000    1.001    1.001 /home/user/test.py:10(b)</code></pre><p>You should be able to make sense of the values in above table.</p><p>Let’s use a tool called <a href="https://jiffyclub.github.io/snakeviz/" target="_blank" rel="noopener">SnakeViz</a> to visualize this. Install the tool and run:</p><pre class=" language-bash"><code class="language-bash">pip <span class="token function">install</span> snakevizsnakeviz cProfile.stats</code></pre><p>You should see the page opened in the browser:</p><p><img src="/images/2024-analyzing-slow-python-code-using-cprofile/analyzing-slow-python-code-using-cprofile-04.png" alt=""></p><p>This is very similar to our diagrams above. Note that <code>sleep()</code> gets merged into a single row at the bottom which shows its <code>tottime</code>.</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;Python is widely considered to be a slow language. However it’s slow only when compared to other fast languages. I’d argue it’s a perfect
      
    
    </summary>
    
    
      <category term="Python" scheme="https://www.sheshbabu.com/tags/Python/"/>
    
      <category term="Performance" scheme="https://www.sheshbabu.com/tags/Performance/"/>
    
  </entry>
  
  <entry>
    <title>Thoughts on the Future of Software Development</title>
    <link href="https://www.sheshbabu.com/posts/thoughts-on-the-future-of-software-development/"/>
    <id>https://www.sheshbabu.com/posts/thoughts-on-the-future-of-software-development/</id>
    <published>2024-03-18T09:57:18.000Z</published>
    <updated>2024-05-19T06:17:28.272Z</updated>
    
    <content type="html"><![CDATA[<p>Large Language Models (LLMs) caused a huge stir in the creative circles when they were able to generate images, text and code. Initially the results were quite hilarious with <a href="https://twitter.com/weirddalle/status/1617240690290839553" target="_blank" rel="noopener">drawings of people with messed up hands</a>, hallucinating incorrect facts and code. But things are slowly <em>and steadily</em> getting better. Before the advent of these models, the main argument against automating these tasks was that machines can’t think creatively. Now this argument gets weaker by the day. Where do we go from here?</p><p>The downside of trying to think about some vague problem like predicting future is that your thoughts get muddled and it’s hard to think clearly. So we need to come up with frameworks and analogies for us to lean on. </p><p><img src="/images/2024-thoughts-on-the-future-of-software-development/2024-thoughts-on-the-future-of-software-development.png" alt=""></p><h2 id="Framework-Software-Development-Capability-Level"><a href="#Framework-Software-Development-Capability-Level" class="headerlink" title="Framework: Software Development Capability Level"></a>Framework: Software Development Capability Level</h2><p>Software development is not just about writing code. The image people have of programmers is a person sitting in a dark room looking at computers and furiously typing code. Though coding all day sounds very appealing, most of software development time is spent on communicating with other people or other admin work instead of just writing code: </p><ul><li>Gathering requirements from business users</li><li>Refining these requirements so they can be modeled as code</li><li>Talking with other team members like Designers and Product Managers to visualize the solution and coming up with a plan of attack</li><li>Working with other developers to come up with a technical design and refining it</li><li>Setting up infrastructure, configuration, boilerplate etc.</li><li>Actually writing some code</li><li>Debugging, trying to understand other people’s code, writing documentation, etc.</li><li>Deploying to production</li><li>Firefighting production issues</li><li>… and so many more tasks</li></ul><p>So saying things like “AI will replace Developers” requires “AI” to be competent in all the above tasks and not just writing code.</p><blockquote><p>So saying things like “AI will replace Developers” requires “AI” to be competent in all the above tasks and not just writing code.</p></blockquote><p>But looking at the list above, it looks like some of these tasks can also be automated in future but not yet. How do we frame this thought? </p><p>The world of self-driving cars come up with a way to <a href="https://en.wikipedia.org/wiki/Self-driving_car#Classifications" target="_blank" rel="noopener">classify the level of automation</a>. It has discrete levels and goes all the way from no automation to partial automation to full automation. I find this very useful for many reasons:</p><ul><li>It clearly describes what the current technology is capable of</li><li>It prevents us from thinking in black and white - it’s not about human driver vs AI driver where AI fully replaces human drivers, it’s possible to have gray areas where human drivers are assisted by AI in things like emergency braking, lane centering etc.</li></ul><p>How does such classification look like to AI driven software development?</p><ul><li>The lowest tier would be what we had before - no AI involved in work. Of course we had other types of automations like compilers, build processes, etc., but these are not AI, these were human written deterministic automation.</li><li>The next level is what we have now - developers using ChatGPT or GitHub Copilot to assist them. They use it for things like writing tests, boilerplate code, refactoring, understanding code/errors etc. It’s like talking with a fellow developer over chat whom you can ask questions and get some help from, but they don’t have access to your machine so they can’t create files, run build commands or deploy to production.</li><li>The highest level would be like delegating part of your project or the entire project to a developer. These “AI coders” would take in the requirements, write the code, fix errors and deploy the final product to production. I assumed were still many months away from this happening, but was proved wrong with the <a href="https://www.youtube.com/watch?v=fjHtjT7GO1c" target="_blank" rel="noopener">Devin demo</a> - even though it can only <a href="https://favtutor.com/articles/devin-ai-early-insights/" target="_blank" rel="noopener">perform simple development tasks</a> now, there’s a chance that this will improve in future.</li></ul><p><img src="/images/2024-thoughts-on-the-future-of-software-development/2024-thoughts-on-the-future-of-software-development-01.png" alt=""></p><p>Apart from what the AI model is capable of, we should ask think in terms of how accurate the solutions are. Initially these models were prone to hallucinations or you need to prompt them in specific ways to get what you want. This adds friction to adoption and most people give up on AI assistants at this point. But this is also improving, the newer models don’t need that level of prompt engineering. Also, the models should be able to “learn” by browsing the web instead of relying on their training data. This is important as new versions of libraries and programming languages get introduced. </p><h2 id="Framework-Outsourced-Software-Development"><a href="#Framework-Outsourced-Software-Development" class="headerlink" title="Framework: Outsourced Software Development"></a>Framework: Outsourced Software Development</h2><p>Now that we’ve established the capabilities, how would these influence the team or organizational structure? Companies do software development in multiple ways:</p><ul><li>Fully inhouse</li><li>Mostly inhouse with few vendors</li><li>Mostly vendors with few inhouse</li><li>Fully vendors</li></ul><p><img src="/images/2024-thoughts-on-the-future-of-software-development/2024-thoughts-on-the-future-of-software-development-02.png" alt=""></p><p>In a way, we can think of AI coders as outsourced software vendors/consultants. Some companies use them a lot and some don’t as much. Irrespective of the composition, I believe it’s always important to have an inhouse team oversee their work. This is to make sure vendor’s output is aligned with your organization’s long term goals. Of course, you can solve this via contracts, but they usually only apply to a specific vendor or a project, and you can’t enforce long terms goals using this method. It’s always better to have at least a small inhouse team who can guide the vendors. Similarly, even when AI coders can be rented out like EC2 instances, it will be beneficial to have an inhouse team of Software Developers to oversee their work.</p><h2 id="Framework-Software-Development-Is-Modeling-Complexity"><a href="#Framework-Software-Development-Is-Modeling-Complexity" class="headerlink" title="Framework: Software Development Is Modeling Complexity"></a>Framework: Software Development Is Modeling Complexity</h2><p>If we’re talking about solving business problems, let’s take sometime to talk about the elephant in the room - Excel. It’s a well known secret that the world runs on Excel, and <a href="https://news.microsoft.com/speeches/satya-nadella-and-terry-myerson-build-2016/" target="_blank" rel="noopener">more than 1 Billion people</a> use it. It provides a very low barrier to entry for business users who want to organize data, perform data analysis, or automate some process. However, we can’t use Excel for complex business workflows as it doesn’t have features like granular access control, ability to integrate with unsupported systems, testability, reusability, or just vendor lock-in etc. The same can be said for “Low Code” solutions like <a href="https://www.microsoft.com/en-us/power-platform/products/power-automate" target="_blank" rel="noopener">Power Automate</a>, etc.</p><p>Coming back to the original question, would business users be able to use AI coders to create these complex workflows without the help of software developers?</p><blockquote><p>Would business users be able to use AI coding tools to create these complex workflows without the help of software developers?</p></blockquote><p>If you think about it, Excel and Low Code tools have existed for many decades, so why does the Software Development profession still exist? It goes back to thinking of Software Development as just writing code. For complex problems, we need people who can effectively manage these complexity and translate the business problems from real world domain to digital models. </p><p>In other words, if you’re able to build a wooden shed from YouTube tutorials without the help of a Civil Engineer, doesn’t mean you can/should do the same for a 10 story building. If you go about learning how to do this properly, then you slowly become a Civil Engineer! It’s just a matter of whether you’re willing to put in the time to learn this properly or hire an experienced Engineer to do it for you. </p><p>So whether these people are using Excel or the latest AI coder, if they’re modeling complex logic, they’re still software developers in my opinion! They’re just using different tools to express the business requirements - spreadsheet formulas vs code vs prompts. </p><h2 id="Framework-Size-Of-The-Pie"><a href="#Framework-Size-Of-The-Pie" class="headerlink" title="Framework: Size Of The Pie"></a>Framework: Size Of The Pie</h2><p>Most of the anxiety surrounding this topic assumes the size of the market for Software Development remains the same - AI coders will slowly take “market share” away from humans. </p><p><img src="/images/2024-thoughts-on-the-future-of-software-development/2024-thoughts-on-the-future-of-software-development-03.png" alt=""></p><p>From the previous section, we know that the market size of “solving business problems” is much much bigger than just Software Development. So there’s no reason to believe that Software Development will disappear anytime soon.</p><p><img src="/images/2024-thoughts-on-the-future-of-software-development/2024-thoughts-on-the-future-of-software-development-04.png" alt=""></p><h2 id="Framework-Formal-Business-Logic-Definition"><a href="#Framework-Formal-Business-Logic-Definition" class="headerlink" title="Framework: Formal Business Logic Definition"></a>Framework: Formal Business Logic Definition</h2><p>Business logic must always be defined in an unambiguous format. This is why programming languages, even though they use English words like “if”, “switch” etc., are very particular about what these words mean and won’t work if you use the wrong words. If you think about it, the same goes for Excel formulas or Low Code flows. </p><p>In future, even if the AI coders could generate a software product from instructions given in conversational English, I believe there would still be an underlying formal definition of the business logic generated in the backend. It might look very different from the languages and frameworks we use today, but a formal definition of business logic sounds a lot like “code”. </p><p>Until AI coders can start generating these business logic from conversational English in a deterministic manner, there would still be a need for people who can understand the code it generates in the backend and make changes if necessary. These people would be Software Developers. </p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>In summary, I believe there would still be a market for Software Developers in the foreseeable future, though the nature of work will change and the tools we would use might be very different from what we have now.</p>]]></content>
    
    <summary type="html">
    
      My optimistic take on the future of Software Development in the age of AI.
    
    </summary>
    
    
      <category term="Development" scheme="https://www.sheshbabu.com/tags/Development/"/>
    
      <category term="LLM" scheme="https://www.sheshbabu.com/tags/LLM/"/>
    
  </entry>
  
  <entry>
    <title>System-wide text summarization using Ollama and AppleScript</title>
    <link href="https://www.sheshbabu.com/posts/system-wide-text-summarization-using-ollama-and-applescript/"/>
    <id>https://www.sheshbabu.com/posts/system-wide-text-summarization-using-ollama-and-applescript/</id>
    <published>2024-03-11T01:06:21.000Z</published>
    <updated>2024-07-20T10:04:51.910Z</updated>
    
    <content type="html"><![CDATA[<p>Local LLMs like Mistral, Llama etc allow us to run ChatGPT like large language models locally inside our computers. Since all the processing happens within our systems, I feel more comfortable feeding it personal data compared to hosted LLMs.</p><p>A quick way to get started with Local LLMs is to use an application like <a href="https://ollama.com/" target="_blank" rel="noopener">Ollama</a>. It’s very easy to install, but interacting with it involves <a href="https://github.com/ollama/ollama?tab=readme-ov-file#quickstart" target="_blank" rel="noopener">running commands</a> on a terminal or installing other <a href="https://github.com/open-webui/open-webui" target="_blank" rel="noopener">server based GUI</a> in your system.</p><p>These are not a huge dealbreakers, but wouldn’t it be nice if you can select a piece of text in any application and ask the LLM to summarize it?</p><blockquote><p>Wouldn’t it be nice if you can select a piece of text in any application and ask the LLM to summarize it?</p></blockquote><p>In order to implement this, we would need to figure out a way to pass the current selected text to the ollama CLI program.</p><p>In macOS, there’s a concept call “<a href="https://developer.apple.com/library/archive/documentation/LanguagesUtilities/Conceptual/MacAutomationScriptingGuide/MakeaSystem-WideService.html" target="_blank" rel="noopener">Services</a>” which allows you to pass data from one app to another. For example. if you select a piece of text in most applications and right click, you would see this “Services” menu.</p><p><img src="/images/2024-ollama-and-applescript/2024-ollama-and-applescript-01.png" alt=""></p><p>We can use this feature to send data to the ollama CLI program.</p><p>To implement this, we need to use a builtin macOS application called “Automator”. Simply launch Automator, select “New Document” in the file picker dialog and choose “Quick Action” as the document type.</p><p><img src="/images/2024-ollama-and-applescript/2024-ollama-and-applescript-02.png" alt=""></p><p>You should see something like the above.</p><p>Add “Run Shell Script” and “Run AppleScript” actions as shown in the below screenshot and copy paste the following into them:</p><pre class=" language-shell"><code class="language-shell">/usr/local/bin/ollama run mistral summarize:</code></pre><pre class=" language-applescript"><code class="language-applescript"><span class="token keyword">on</span> run <span class="token punctuation">{</span>input<span class="token punctuation">,</span> parameters<span class="token punctuation">}</span>    display dialog <span class="token punctuation">(</span>input <span class="token operator">as</span> <span class="token class builtin">text</span><span class="token punctuation">)</span>    <span class="token keyword">return</span> input<span class="token keyword">end</span> run</code></pre><p><img src="/images/2024-ollama-and-applescript/2024-ollama-and-applescript-03.png" alt=""></p><p>Save this Quick Action as “Summarize with LLM” and you should see it in the Services menu.</p><p><img src="/images/2024-ollama-and-applescript/2024-ollama-and-applescript-04.png" alt=""></p><p>Let’s try this out! I select the content of a <a href="http://localhost:4000/posts/rust-module-system/" target="_blank" rel="noopener">blog post</a> and choose “Summarize with LLM”. After a few seconds, I will see a summarized version of the selected content.</p><p><img src="/images/2024-ollama-and-applescript/2024-ollama-and-applescript-05.png" alt=""></p><p>This Quick Action can be used in most applications, try it in your notes, email, etc!</p><p>Keep in mind that the first call to this Quick Action would take many minutes as the LLM needs to be downloaded into your machine. Subsequent invocations should only take a few seconds depending on how fast your machine is.</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;Local LLMs like Mistral, Llama etc allow us to run ChatGPT like large language models locally inside our computers. Since all the process
      
    
    </summary>
    
    
      <category term="LLM" scheme="https://www.sheshbabu.com/tags/LLM/"/>
    
      <category term="AppleScript" scheme="https://www.sheshbabu.com/tags/AppleScript/"/>
    
  </entry>
  
  <entry>
    <title>Entity Resolution Challenges</title>
    <link href="https://www.sheshbabu.com/posts/entity-resolution-challenges/"/>
    <id>https://www.sheshbabu.com/posts/entity-resolution-challenges/</id>
    <published>2023-08-01T22:06:45.000Z</published>
    <updated>2024-05-19T06:17:28.270Z</updated>
    
    <content type="html"><![CDATA[<p>Entity Resolution (also known as Record Linkage, Deduplication etc) is the process of identifying the same entity (person, company, product etc) across one or more datasets and combining them into a single record.</p><p>For example, if you have the following datasets containing information about companies:</p><p><img src="/images/2023-entity-resolution-challenges/2023-entity-resolution-challenges-02.png" alt=""></p><p>You’ll be able to get better insights if you combine the data from both the datasets instead of looking at each dataset in isolation.</p><p><img src="/images/2023-entity-resolution-challenges/2023-entity-resolution-challenges-03.png" alt=""></p><h2 id="No-unique-identifiers"><a href="#No-unique-identifiers" class="headerlink" title="No unique identifiers"></a>No unique identifiers</h2><p>You might be wondering what makes this process challenging - can’t we just do a SQL join on the datasets based on a unique identifier like SSN (for people) or stock tickers (for companies)?</p><p>Unfortunately, not all datasets come with unique identifiers. Even if they do have unique identifiers, it might be that they are different - one dataset might contain SSN and other might have University Student ID.</p><p>Attributes like person’s name, date of birth etc can’t be used as unique identifiers either.</p><p>For example, we normally assume date of birth to be somewhat unique among the population and that it’s rare for 2 people to have the same birthday. There are 365 days in a year, so there’s only 365 unique date of births - it’s actually not that unique. Even so, we intuitively think that we need a group of at least 365 people for 2 of them to have the same birthday. But if we <a href="https://en.wikipedia.org/wiki/Birthday_problem#Calculating_the_probability" target="_blank" rel="noopener">do the math</a>, we see that we only need a group of 70 people for 2 of them to have 99.9% chance of same birthday! Birthdays are even less unique than we thought it was. This is known as the <a href="https://en.wikipedia.org/wiki/Birthday_problem" target="_blank" rel="noopener">birthday paradox</a>, and this holds true for other attributes as well.</p><p>If we don’t have unique identifiers (SSN), and we can’t rely on non-unique identifiers (date of birth), what else can we use?</p><p>We can use a combination of non-unique identifiers as a “composite identifier”. For example, we can use person’s <code>first name + last name + date of birth + gender</code> as a composite identifier.</p><h2 id="Conflicting-data"><a href="#Conflicting-data" class="headerlink" title="Conflicting data"></a>Conflicting data</h2><p>While this approach make sense, there are still some limitations depending on the data sources you use:</p><p><strong>Outdated data</strong></p><p>People change their names, their addresses etc. Companies might have gone through mergers or acquisitions which could have resulted in significant changes to the data. These changes might be captured in one data source but not in others.</p><p><strong>Inconsistent data</strong></p><p>Depending on the way the data is gathered, someone might have entered their nickname in one data source but their full name in another. One data source might contain the company’s legal name but the other might have DBA (Doing Business As) name. If there’s no mapping available between these alternative representations, then it’s hard to reconcile.</p><p><strong>Missing data</strong></p><p>Important attributes that could’ve been part of the composite identifier might have a lot of NULL values in either of the datasets.</p><p><strong>Dirty data</strong></p><p>There could be typos and other mistakes while gathering the data.</p><p><strong>Fake data</strong></p><p>Some users might intentionally provide wrong information.</p><p><img src="/images/2023-entity-resolution-challenges/2023-entity-resolution-challenges-06.png" alt=""></p><p>These inconsistencies reduce the effectiveness of using the composite identifiers, but most of the time, they’re all we would have to work with.</p><h2 id="Scalability"><a href="#Scalability" class="headerlink" title="Scalability"></a>Scalability</h2><p>Aside from data quality issues, the size of the datasets also pose a challenge. We need to do a pairwise comparison between the records of the two datasets and decide whether they match or not.</p><p><img src="/images/2023-entity-resolution-challenges/2023-entity-resolution-challenges-05.gif" alt=""></p><p>This pairwise comparison scales quadratically - if you have <code>m</code> number of records in one dataset and <code>n</code> number of records in the other, then the number of comparisons is <code>m * n</code>. If you remove duplicates, it becomes <code>(m * n) / 2</code>.</p><p>Big O notation doesn’t care about exact values and is only concerned about how an algorithm scales with the size of data. So we make <code>m</code> and <code>n</code> the same and remove the constant <code>1/2</code>. The number of comparisons become <code>n * n</code> which is <code>n²</code> or quadratic complexity.</p><p><img src="/images/2023-entity-resolution-challenges/2023-entity-resolution-challenges-01.png" alt=""></p><p>In order to give a sense of how bad this is, let’s say the <code>n</code> is 100,000 records. If the comparison algorithm takes about 0.05 milliseconds, then <code>100,000 * 100,000 comparisons * 0.05 milliseconds = 5.79 days</code>!</p><table><thead><tr><th>Number of Records</th><th>Time Taken</th></tr></thead><tbody><tr><td>1000</td><td>50 seconds</td></tr><tr><td>10,000</td><td>1.3 hours</td></tr><tr><td>100,000</td><td>5.8 days</td></tr><tr><td>1,000,000</td><td>1.6 years</td></tr><tr><td>10,000,000</td><td>158 years</td></tr></tbody></table><h3 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h3><p>Depending on the datasets you’re dealing with, entity resolution task can be as trivial as an SQL join (if you have unique identifiers and good quality data), or a comfortable challenge (large scale and good quality data), or an impossibly wicked obstacle to overcome (large scale and bad quality data).</p><p><img src="/images/2023-entity-resolution-challenges/2023-entity-resolution-challenges-04.png" alt=""></p><p>Luckily this is not a novel problem, very smart people around the world have spent many years trying to solve this problem. They have come up with many solutions for solving the above challenges. I will go through some of them in future posts.</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;Entity Resolution (also known as Record Linkage, Deduplication etc) is the process of identifying the same entity (person, company, produ
      
    
    </summary>
    
    
      <category term="Entity Resolution" scheme="https://www.sheshbabu.com/tags/Entity-Resolution/"/>
    
  </entry>
  
  <entry>
    <title>Alerting: Expectation vs Reality</title>
    <link href="https://www.sheshbabu.com/posts/alerting-expectation-vs-reality/"/>
    <id>https://www.sheshbabu.com/posts/alerting-expectation-vs-reality/</id>
    <published>2023-07-22T05:23:10.000Z</published>
    <updated>2024-05-19T06:17:28.270Z</updated>
    
    <content type="html"><![CDATA[<p>Let’s say you’ve a moderately complex system that your team has been working on for the past couple of years. It works well and the users are generally happy. But now-and-then you hear user complaints about system being down or not able to use a feature. You think perhaps you need be more proactive in catching these issues than being reactive to user complaints.</p><p>In your mind, you think the system works well for most of the days, and goes down or causes errors for couple of days a year.</p><p><img src="/images/2023-alerting-expectation-vs-reality/2023-alerting-expectation-vs-reality-01.png" alt=""></p><p>To be more proactive in catching errors, you implement monitoring and alerting for your system, and this is what you see now:</p><p><img src="/images/2023-alerting-expectation-vs-reality/2023-alerting-expectation-vs-reality-02.png" alt=""></p><p>You expected less frequent issues in your system, but looks like your system is on fire every single day!</p><h3 id="Why-is-there-a-big-gap-between-expectation-and-reality"><a href="#Why-is-there-a-big-gap-between-expectation-and-reality" class="headerlink" title="Why is there a big gap between expectation and reality?"></a>Why is there a big gap between expectation and reality?</h3><ul><li>Not all users report errors. Some might stop using your product and turn to alternatives. Only a few dedicated users were giving you feedback before.</li><li>Some of the errors you’re seeing are edge cases that escaped your QA process - there’s only so much manual or automated testing can catch.</li><li>Some errors might be from endpoints or services that are not that critical to the users, so the users didn’t report it before.</li><li>Other errors might be duplicates. If you’re using microservices architecture, then if there’s an error in one service, all the upstream services that depend on it would also throw errors.</li></ul><h3 id="Alert-fatigue"><a href="#Alert-fatigue" class="headerlink" title="Alert fatigue"></a>Alert fatigue</h3><p>One big consequence of dealing with alerts everyday is that your team gets numb to them. This is called <a href="https://en.wikipedia.org/wiki/Alarm_fatigue" target="_blank" rel="noopener">alert fatigue</a>. If everything is on fire all the time, one more fire doesn’t make much of a difference - but this fire can be your authentication system being down and none of your users can login! Not only you’re missing out on alerts, your team also gets mentally tired and unable to focus on other tasks.</p><h3 id="What-can-we-do-about-this"><a href="#What-can-we-do-about-this" class="headerlink" title="What can we do about this?"></a>What can we do about this?</h3><ul><li>Instead of having the same set of people looking into the alerts, rotate people so they have time to recover.</li><li>Triage the issues first. Spend some time understanding the business impact of the error. This would help you prioritize which errors to fix first.</li><li>If the errors are not important, you can silence them either temporarily (for your team to catch up to more important errors) or permanently.</li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;Let’s say you’ve a moderately complex system that your team has been working on for the past couple of years. It works well and the users
      
    
    </summary>
    
    
      <category term="Development" scheme="https://www.sheshbabu.com/tags/Development/"/>
    
      <category term="Infrastructure" scheme="https://www.sheshbabu.com/tags/Infrastructure/"/>
    
      <category term="Microservices" scheme="https://www.sheshbabu.com/tags/Microservices/"/>
    
  </entry>
  
  <entry>
    <title>Exploring Singapore’s OneMap API</title>
    <link href="https://www.sheshbabu.com/posts/exploring-singapore-onemap-api/"/>
    <id>https://www.sheshbabu.com/posts/exploring-singapore-onemap-api/</id>
    <published>2023-06-29T08:07:54.000Z</published>
    <updated>2024-05-19T06:17:28.270Z</updated>
    
    <content type="html"><![CDATA[<p>Lot of people use Google Maps on a regular basis, however when it comes to programmatic access, using Google Maps API can get quite expensive.</p><p>If your mapping requirements are restricted to Singapore, you can use the free OneMap API developed by Singapore Land Authority (SLA). Apart from being free, it’s also considered to have the most accurate, detailed and up-to-date information.</p><p><img src="/images/2023-exploring-singapore-onemap-api/2023-exploring-singapore-onemap-api.png" alt=""></p><h2 id="Authentication"><a href="#Authentication" class="headerlink" title="Authentication"></a>Authentication</h2><p>Some endpoints such as geocoding don’t require authentication but others like routing require it. The authentication flow is a bit unconventional, here’s how it works:</p><p><strong>Initial registration</strong></p><ul><li>Register for an account <a href="https://developers.onemap.sg/register/" target="_blank" rel="noopener">here</a></li><li>Once registered, use the confirmation code sent via email <a href="https://developers.onemap.sg/confirm_account/" target="_blank" rel="noopener">here</a></li></ul><p><strong>Authenticated requests</strong></p><ul><li>Hit the <code>getToken</code> endpoint to generate new tokens</li><li>This endpoint takes in your email and password set during registration</li><li>The response contains an access token (JWT) with a TTL of 3 days</li><li>It’s not mentioned in the API docs, but this looks similar to the <a href="https://www.rfc-editor.org/rfc/rfc6749#section-1.3.3" target="_blank" rel="noopener">OAuth ROPC Flow</a></li></ul><p>Here’s how the <code>getToken</code> request and response looks like:</p><pre class=" language-shell"><code class="language-shell">curl --request POST \  --url https://developers.onemap.sg/privateapi/auth/post/getToken \  --header 'Content-Type: application/json' \  --data '{  "email": "abc@def.com",  "password": "hunter2"}'</code></pre><pre class=" language-json"><code class="language-json"><span class="token punctuation">{</span>  <span class="token property">"access_token"</span><span class="token operator">:</span> <span class="token string">"--token--"</span><span class="token punctuation">,</span>  <span class="token property">"expiry_timestamp"</span><span class="token operator">:</span> <span class="token string">"--expiry--"</span><span class="token punctuation">}</span></code></pre><p>If you’re using JavaScript, you can store the token in an in-memory cache, and use something like an <a href="https://axios-http.com/docs/interceptors" target="_blank" rel="noopener">Axios request interceptor</a> to check for token expiry and either use the cache or fetch new token for authenticated requests. Same can be implemented using other libraries like <a href="https://www.python-httpx.org/advanced/#event-hooks" target="_blank" rel="noopener">HTTPX</a></p><h2 id="Geocoding"><a href="#Geocoding" class="headerlink" title="Geocoding"></a>Geocoding</h2><p><a href="https://en.wikipedia.org/wiki/Address_geocoding" target="_blank" rel="noopener">Geocoding</a> is the process of converting an address into map coordinates like latitude and longitude.</p><p>We can use the <code>search</code> endpoint to get the coordinates for a location, building or postal code. Here’s a query for “novena mrt”</p><pre class=" language-shell"><code class="language-shell">curl --request GET \  --url 'https://developers.onemap.sg/commonapi/search?searchVal=novena%20mrt&returnGeom=Y&getAddrDetails=N'</code></pre><pre class=" language-json"><code class="language-json"><span class="token punctuation">{</span>  <span class="token property">"found"</span><span class="token operator">:</span> <span class="token number">6</span><span class="token punctuation">,</span>  <span class="token property">"totalNumPages"</span><span class="token operator">:</span> <span class="token number">1</span><span class="token punctuation">,</span>  <span class="token property">"pageNum"</span><span class="token operator">:</span> <span class="token number">1</span><span class="token punctuation">,</span>  <span class="token property">"results"</span><span class="token operator">:</span> <span class="token punctuation">[</span>    <span class="token punctuation">{</span>      <span class="token property">"SEARCHVAL"</span><span class="token operator">:</span> <span class="token string">"NOVENA MRT STATION (NS20)"</span><span class="token punctuation">,</span>      <span class="token property">"X"</span><span class="token operator">:</span> <span class="token string">"29169.3297925005"</span><span class="token punctuation">,</span>      <span class="token property">"Y"</span><span class="token operator">:</span> <span class="token string">"33633.1535391063"</span><span class="token punctuation">,</span>      <span class="token property">"LATITUDE"</span><span class="token operator">:</span> <span class="token string">"1.32044079120154"</span><span class="token punctuation">,</span>      <span class="token property">"LONGITUDE"</span><span class="token operator">:</span> <span class="token string">"103.843825618748"</span><span class="token punctuation">,</span>      <span class="token property">"LONGTITUDE"</span><span class="token operator">:</span> <span class="token string">"103.843825618748"</span>    <span class="token punctuation">}</span><span class="token punctuation">,</span>    ...  <span class="token punctuation">]</span><span class="token punctuation">}</span></code></pre><p>Notice how we get both X/Y coordinates in addition to the LATITUDE/LONGITUDE values. The X/Y coordinates (SVY21) can provide <a href="https://app.sla.gov.sg/sirent/About/PlaneCoordinateSystem" target="_blank" rel="noopener">more accurate representation</a> of the area compared to the LATITUDE/LONGITUDE values (WGS84).</p><p>Keep in mind that this endpoint is not <a href="https://www.algolia.com/doc/guides/managing-results/optimize-search-results/typo-tolerance/" target="_blank" rel="noopener">typo tolerant</a> so using the correct spelling is important.</p><h2 id="Reverse-Geocoding"><a href="#Reverse-Geocoding" class="headerlink" title="Reverse Geocoding"></a>Reverse Geocoding</h2><p>Reverse geocoding is, well, doing the reverse of geocoding - if you provide latitude/longitude, you get the address. We can either pass in the X/Y coordinates (SVY21) or LATITUDE/LONGITUDE coordinates (WGS84)</p><p>Using the X, Y values from the above search for “novena mrt”</p><pre class=" language-shell"><code class="language-shell">curl --request GET \  --url 'https://developers.onemap.sg/privateapi/commonsvc/revgeocodexy?location=29169.3297925005%2C33633.1535391063&token=--token--'</code></pre><p>Same for the LATITUDE/LONGITUDE values:</p><pre class=" language-shell"><code class="language-shell">curl --request GET \  --url 'https://developers.onemap.sg/privateapi/commonsvc/revgeocode?location=1.32044079120154%2C103.843825618748&token=--token--'</code></pre><pre class=" language-json"><code class="language-json"><span class="token punctuation">{</span>  <span class="token property">"GeocodeInfo"</span><span class="token operator">:</span> <span class="token punctuation">[</span>    <span class="token punctuation">{</span>      <span class="token property">"BUILDINGNAME"</span><span class="token operator">:</span> <span class="token string">"NOVENA SQUARE"</span><span class="token punctuation">,</span>      <span class="token property">"BLOCK"</span><span class="token operator">:</span> <span class="token string">"238"</span><span class="token punctuation">,</span>      <span class="token property">"ROAD"</span><span class="token operator">:</span> <span class="token string">"THOMSON ROAD"</span><span class="token punctuation">,</span>      <span class="token property">"POSTALCODE"</span><span class="token operator">:</span> <span class="token string">"307683"</span><span class="token punctuation">,</span>      <span class="token property">"XCOORD"</span><span class="token operator">:</span> <span class="token string">"29172.1136298"</span><span class="token punctuation">,</span>      <span class="token property">"YCOORD"</span><span class="token operator">:</span> <span class="token string">"33580.8177133"</span><span class="token punctuation">,</span>      <span class="token property">"LATITUDE"</span><span class="token operator">:</span> <span class="token string">"1.319967484392001"</span><span class="token punctuation">,</span>      <span class="token property">"LONGITUDE"</span><span class="token operator">:</span> <span class="token string">"103.84385063099295"</span><span class="token punctuation">,</span>      <span class="token property">"LONGTITUDE"</span><span class="token operator">:</span> <span class="token string">"103.84385063099295"</span>    <span class="token punctuation">}</span><span class="token punctuation">,</span>    <span class="token punctuation">{</span>      <span class="token property">"BUILDINGNAME"</span><span class="token operator">:</span> <span class="token string">"NOVENA SQUARE"</span><span class="token punctuation">,</span>      <span class="token property">"BLOCK"</span><span class="token operator">:</span> <span class="token string">"238A"</span><span class="token punctuation">,</span>      <span class="token property">"ROAD"</span><span class="token operator">:</span> <span class="token string">"THOMSON ROAD"</span><span class="token punctuation">,</span>      <span class="token property">"POSTALCODE"</span><span class="token operator">:</span> <span class="token string">"307684"</span><span class="token punctuation">,</span>      <span class="token property">"XCOORD"</span><span class="token operator">:</span> <span class="token string">"29172.1136298"</span><span class="token punctuation">,</span>      <span class="token property">"YCOORD"</span><span class="token operator">:</span> <span class="token string">"33580.8177133"</span><span class="token punctuation">,</span>      <span class="token property">"LATITUDE"</span><span class="token operator">:</span> <span class="token string">"1.319967484392001"</span><span class="token punctuation">,</span>      <span class="token property">"LONGITUDE"</span><span class="token operator">:</span> <span class="token string">"103.84385063099295"</span><span class="token punctuation">,</span>      <span class="token property">"LONGTITUDE"</span><span class="token operator">:</span> <span class="token string">"103.84385063099295"</span>    <span class="token punctuation">}</span><span class="token punctuation">,</span>    <span class="token punctuation">{</span>      <span class="token property">"BUILDINGNAME"</span><span class="token operator">:</span> <span class="token string">"NOVENA MRT STATION"</span><span class="token punctuation">,</span>      <span class="token property">"BLOCK"</span><span class="token operator">:</span> <span class="token string">"250"</span><span class="token punctuation">,</span>      <span class="token property">"ROAD"</span><span class="token operator">:</span> <span class="token string">"THOMSON ROAD"</span><span class="token punctuation">,</span>      <span class="token property">"POSTALCODE"</span><span class="token operator">:</span> <span class="token string">"307642"</span><span class="token punctuation">,</span>      <span class="token property">"XCOORD"</span><span class="token operator">:</span> <span class="token string">"29171.2246894"</span><span class="token punctuation">,</span>      <span class="token property">"YCOORD"</span><span class="token operator">:</span> <span class="token string">"33635.0957146"</span><span class="token punctuation">,</span>      <span class="token property">"LATITUDE"</span><span class="token operator">:</span> <span class="token string">"1.3204583554775822"</span><span class="token punctuation">,</span>      <span class="token property">"LONGITUDE"</span><span class="token operator">:</span> <span class="token string">"103.84384264546134"</span><span class="token punctuation">,</span>      <span class="token property">"LONGTITUDE"</span><span class="token operator">:</span> <span class="token string">"103.84384264546134"</span>    <span class="token punctuation">}</span>  <span class="token punctuation">]</span><span class="token punctuation">}</span></code></pre><p>Makes sense as both the Novena Mall and the MRT station are in the same location.</p><h2 id="Routing"><a href="#Routing" class="headerlink" title="Routing"></a>Routing</h2><p>OneMap also provides a way to get directions between two locations. Let’s plan a simple route from Thomson Medical Centre (1.32536064328127, 103.841457941408) to Novena MRT (1.3204583554775822, 103.84384264546134) by driving:</p><pre class=" language-shell"><code class="language-shell">curl --request GET \  --url 'https://developers.onemap.sg/privateapi/routingsvc/route?start=1.32536064328127%2C103.841457941408&end=1.3204583554775822%2C103.84384264546134&routeType=drive&token=--token--'</code></pre><pre class=" language-json"><code class="language-json"><span class="token punctuation">{</span>  <span class="token property">"status_message"</span><span class="token operator">:</span> <span class="token string">"Found route between points"</span><span class="token punctuation">,</span>  <span class="token property">"route_geometry"</span><span class="token operator">:</span> <span class="token string">"qxaGqqxxRA{@eAA_ABm@Bg@Jc@JKBFGBBb@Kf@Kl@C~@CdA@bBLJ@bAJhBLb@DL@t@FF?\\BtAHj@?J?XAvA_@`@Kd@QJE??DC~@i@f@_@h@a@bBqAz@m@"</span><span class="token punctuation">,</span>  <span class="token property">"status"</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">,</span>  <span class="token property">"route_instructions"</span><span class="token operator">:</span> <span class="token punctuation">[</span>    <span class="token punctuation">[</span>      <span class="token string">"Left"</span><span class="token punctuation">,</span>      <span class="token string">"THOMSON ROAD"</span><span class="token punctuation">,</span>      <span class="token number">34</span><span class="token punctuation">,</span>      <span class="token string">"1.32505,103.841685"</span><span class="token punctuation">,</span>      <span class="token number">11</span><span class="token punctuation">,</span>      <span class="token string">"34m"</span><span class="token punctuation">,</span>      <span class="token string">"East"</span><span class="token punctuation">,</span>      <span class="token string">"North"</span><span class="token punctuation">,</span>      <span class="token string">"driving"</span><span class="token punctuation">,</span>      <span class="token string">"Head East On Thomson Road"</span>    <span class="token punctuation">]</span><span class="token punctuation">,</span>    <span class="token punctuation">[</span>      <span class="token string">"Left"</span><span class="token punctuation">,</span>      <span class="token string">"THOMSON ROAD"</span><span class="token punctuation">,</span>      <span class="token number">145</span><span class="token punctuation">,</span>      <span class="token string">"1.325062,103.841987"</span><span class="token punctuation">,</span>      <span class="token number">17</span><span class="token punctuation">,</span>      <span class="token string">"145m"</span><span class="token punctuation">,</span>      <span class="token string">"North"</span><span class="token punctuation">,</span>      <span class="token string">"East"</span><span class="token punctuation">,</span>      <span class="token string">"driving"</span><span class="token punctuation">,</span>      <span class="token string">"Turn Left To Stay On Thomson Road"</span>    <span class="token punctuation">]</span><span class="token punctuation">,</span>    <span class="token punctuation">[</span>      <span class="token string">"Uturn"</span><span class="token punctuation">,</span>      <span class="token string">"THOMSON ROAD"</span><span class="token punctuation">,</span>      <span class="token number">817</span><span class="token punctuation">,</span>      <span class="token string">"1.326342,103.841838"</span><span class="token punctuation">,</span>      <span class="token number">102</span><span class="token punctuation">,</span>      <span class="token string">"817m"</span><span class="token punctuation">,</span>      <span class="token string">"North"</span><span class="token punctuation">,</span>      <span class="token string">"North"</span><span class="token punctuation">,</span>      <span class="token string">"driving"</span><span class="token punctuation">,</span>      <span class="token string">"Make A U-turn And Continue On Thomson Road"</span>    <span class="token punctuation">]</span><span class="token punctuation">,</span>    <span class="token punctuation">[</span>      <span class="token string">"Left"</span><span class="token punctuation">,</span>      <span class="token string">"THOMSON ROAD"</span><span class="token punctuation">,</span>      <span class="token number">0</span><span class="token punctuation">,</span>      <span class="token string">"1.319661,103.843181"</span><span class="token punctuation">,</span>      <span class="token number">0</span><span class="token punctuation">,</span>      <span class="token string">"0m"</span><span class="token punctuation">,</span>      <span class="token string">"North"</span><span class="token punctuation">,</span>      <span class="token string">"South East"</span><span class="token punctuation">,</span>      <span class="token string">"driving"</span><span class="token punctuation">,</span>      <span class="token string">"You Have Arrived At Your Destination, On The Left"</span>    <span class="token punctuation">]</span>  <span class="token punctuation">]</span><span class="token punctuation">,</span>  <span class="token property">"route_name"</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token string">"THOMSON ROAD"</span><span class="token punctuation">]</span><span class="token punctuation">,</span>  <span class="token property">"route_summary"</span><span class="token operator">:</span> <span class="token punctuation">{</span>    <span class="token property">"start_point"</span><span class="token operator">:</span> <span class="token string">"THOMSON ROAD"</span><span class="token punctuation">,</span>    <span class="token property">"end_point"</span><span class="token operator">:</span> <span class="token string">"THOMSON ROAD"</span><span class="token punctuation">,</span>    <span class="token property">"total_time"</span><span class="token operator">:</span> <span class="token number">129</span><span class="token punctuation">,</span>    <span class="token property">"total_distance"</span><span class="token operator">:</span> <span class="token number">995</span>  <span class="token punctuation">}</span><span class="token punctuation">,</span>  <span class="token property">"viaRoute"</span><span class="token operator">:</span> <span class="token string">"THOMSON ROAD"</span><span class="token punctuation">,</span>  <span class="token property">"subtitle"</span><span class="token operator">:</span> <span class="token string">"Shortest distance"</span><span class="token punctuation">}</span></code></pre><p>Here’s a screenshot from Google Maps, the <a href="https://goo.gl/maps/Myg3sSVrd4tXpKnv9" target="_blank" rel="noopener">instructions</a> are identical!</p><p><img src="/images/2023-exploring-singapore-onemap-api/2023-exploring-singapore-onemap-api-01.png" alt=""></p><p>You can set other modes apart from driving like walk, pt, cycle using the <code>routeType</code> param. “pt” here stands for public transport which when used unlocks other params like <code>maxWalkDistance</code>, <code>mode</code> (TRANSIT, BUS, RAIL), etc.</p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>OneMap is a valuable resource for people building mapping applications in Singapore. There’s a couple of sharp edges like the search not being typo tolerant, unconventional API naming etc, but they pale in comparison to the utility provided.</p><p>Thanks for reading! Feel free to follow me in <a href="https://twitter.com/sheshbabu" target="_blank" rel="noopener">Twitter</a> for more posts like this :)</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;Lot of people use Google Maps on a regular basis, however when it comes to programmatic access, using Google Maps API can get quite expen
      
    
    </summary>
    
    
      <category term="API" scheme="https://www.sheshbabu.com/tags/API/"/>
    
  </entry>
  
  <entry>
    <title>Structured JSON Logging using FastAPI</title>
    <link href="https://www.sheshbabu.com/posts/fastapi-structured-json-logging/"/>
    <id>https://www.sheshbabu.com/posts/fastapi-structured-json-logging/</id>
    <published>2023-06-24T07:25:26.000Z</published>
    <updated>2024-05-19T06:17:28.270Z</updated>
    
    <content type="html"><![CDATA[<p>Let’s start with a simple FastAPI example:</p><pre class=" language-python"><code class="language-python"><span class="token comment" spellcheck="true"># src/main.py</span><span class="token keyword">import</span> uvicorn<span class="token keyword">from</span> fastapi <span class="token keyword">import</span> FastAPIapp <span class="token operator">=</span> FastAPI<span class="token punctuation">(</span><span class="token punctuation">)</span>@app<span class="token punctuation">.</span>get<span class="token punctuation">(</span><span class="token string">"/"</span><span class="token punctuation">)</span><span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">hello</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>    <span class="token keyword">return</span> <span class="token string">"Hello World"</span><span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">"__main__"</span><span class="token punctuation">:</span>    uvicorn<span class="token punctuation">.</span>run<span class="token punctuation">(</span>app<span class="token punctuation">,</span> host<span class="token operator">=</span><span class="token string">"0.0.0.0"</span><span class="token punctuation">,</span> port<span class="token operator">=</span><span class="token number">8000</span><span class="token punctuation">)</span></code></pre><p>When you run this app, you’ll see logs in text format:</p><pre class=" language-plaintext"><code class="language-plaintext">INFO:     Started server process [9529]INFO:     Waiting for application startup.INFO:     Application startup complete.INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)</code></pre><p>If you make a request to this app, you’ll see similar text logs:</p><pre class=" language-plaintext"><code class="language-plaintext">INFO:     127.0.0.1:50824 - "GET / HTTP/1.1" 200 OK</code></pre><p>What if you want logs in a JSON format?</p><h2 id="JSON-formatting"><a href="#JSON-formatting" class="headerlink" title="JSON formatting"></a>JSON formatting</h2><p>Let’s add a logger module:</p><pre class=" language-python"><code class="language-python"><span class="token comment" spellcheck="true"># src/logger.py</span><span class="token keyword">import</span> json<span class="token keyword">import</span> logging<span class="token keyword">from</span> logging <span class="token keyword">import</span> Formatter<span class="token keyword">class</span> <span class="token class-name">JsonFormatter</span><span class="token punctuation">(</span>Formatter<span class="token punctuation">)</span><span class="token punctuation">:</span>    <span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>        super<span class="token punctuation">(</span>JsonFormatter<span class="token punctuation">,</span> self<span class="token punctuation">)</span><span class="token punctuation">.</span>__init__<span class="token punctuation">(</span><span class="token punctuation">)</span>    <span class="token keyword">def</span> <span class="token function">format</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> record<span class="token punctuation">)</span><span class="token punctuation">:</span>        json_record <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span>        json_record<span class="token punctuation">[</span><span class="token string">"message"</span><span class="token punctuation">]</span> <span class="token operator">=</span> record<span class="token punctuation">.</span>getMessage<span class="token punctuation">(</span><span class="token punctuation">)</span>        <span class="token keyword">return</span> json<span class="token punctuation">.</span>dumps<span class="token punctuation">(</span>json_record<span class="token punctuation">)</span>logger <span class="token operator">=</span> logging<span class="token punctuation">.</span>roothandler <span class="token operator">=</span> logging<span class="token punctuation">.</span>StreamHandler<span class="token punctuation">(</span><span class="token punctuation">)</span>handler<span class="token punctuation">.</span>setFormatter<span class="token punctuation">(</span>JsonFormatter<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>logger<span class="token punctuation">.</span>handlers <span class="token operator">=</span> <span class="token punctuation">[</span>handler<span class="token punctuation">]</span>logger<span class="token punctuation">.</span>setLevel<span class="token punctuation">(</span>logging<span class="token punctuation">.</span>DEBUG<span class="token punctuation">)</span></code></pre><p>Here we’re subclassing <code>logging.Formatter</code> and overriding the <code>format</code> method. This method takes in the log record, using which we can construct our JSON log record. We then add our formatter to the Root logger, remove other handlers and import this new logger module into our main application:</p><pre class=" language-diff"><code class="language-diff">  # src/main.py  import uvicorn  from fastapi import FastAPI<span class="token inserted">+ from src.logger import logger</span>  app = FastAPI()  @app.get("/")  async def hello():<span class="token inserted">+     logger.info("Hello")</span>      return "Hello World"  if __name__ == "__main__":<span class="token deleted">-    uvicorn.run(app, host="0.0.0.0", port=8000)</span><span class="token inserted">+    uvicorn.run(app, host="0.0.0.0", port=8000, log_config=None)</span></code></pre><p>Here’s how the logs look now:</p><pre class=" language-json"><code class="language-json"><span class="token punctuation">{</span><span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"Using selector: KqueueSelector"</span><span class="token punctuation">}</span><span class="token punctuation">{</span><span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"Started server process [9981]"</span><span class="token punctuation">}</span><span class="token punctuation">{</span><span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"Waiting for application startup."</span><span class="token punctuation">}</span><span class="token punctuation">{</span><span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"Application startup complete."</span><span class="token punctuation">}</span><span class="token punctuation">{</span><span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)"</span><span class="token punctuation">}</span></code></pre><pre class=" language-json"><code class="language-json"><span class="token punctuation">{</span><span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"Hello"</span><span class="token punctuation">}</span><span class="token punctuation">{</span><span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"127.0.0.1:50896 - \"GET / HTTP/1.1\" 200"</span><span class="token punctuation">}</span></code></pre><p>Not exactly structured logging, but some progress.</p><h2 id="Structured-logging"><a href="#Structured-logging" class="headerlink" title="Structured logging"></a>Structured logging</h2><p>Let’s add a FastAPI middleware to log the requests. You can change the structure of the JSON log record in this method:</p><pre class=" language-python"><code class="language-python"><span class="token comment" spellcheck="true"># src/log_middleware.py</span><span class="token keyword">from</span> starlette<span class="token punctuation">.</span>middleware<span class="token punctuation">.</span>base <span class="token keyword">import</span> BaseHTTPMiddleware<span class="token keyword">from</span> src<span class="token punctuation">.</span>logger <span class="token keyword">import</span> logger<span class="token keyword">class</span> <span class="token class-name">LogMiddleware</span><span class="token punctuation">(</span>BaseHTTPMiddleware<span class="token punctuation">)</span><span class="token punctuation">:</span>    <span class="token keyword">async</span> <span class="token keyword">def</span> <span class="token function">dispatch</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> request<span class="token punctuation">,</span> call_next<span class="token punctuation">)</span><span class="token punctuation">:</span>        response <span class="token operator">=</span> <span class="token keyword">await</span> call_next<span class="token punctuation">(</span>request<span class="token punctuation">)</span>        logger<span class="token punctuation">.</span>info<span class="token punctuation">(</span>            <span class="token string">"Incoming request"</span><span class="token punctuation">,</span>            extra<span class="token operator">=</span><span class="token punctuation">{</span>                <span class="token string">"req"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span> <span class="token string">"method"</span><span class="token punctuation">:</span> request<span class="token punctuation">.</span>method<span class="token punctuation">,</span> <span class="token string">"url"</span><span class="token punctuation">:</span> str<span class="token punctuation">(</span>request<span class="token punctuation">.</span>url<span class="token punctuation">)</span> <span class="token punctuation">}</span><span class="token punctuation">,</span>                <span class="token string">"res"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span> <span class="token string">"status_code"</span><span class="token punctuation">:</span> response<span class="token punctuation">.</span>status_code<span class="token punctuation">,</span> <span class="token punctuation">}</span><span class="token punctuation">,</span>            <span class="token punctuation">}</span><span class="token punctuation">,</span>        <span class="token punctuation">)</span>        <span class="token keyword">return</span> response</code></pre><p>Update the <code>logger.py</code> module to look for these new fields and add them to the final log output. Also, disable the <code>uvicorn.access</code> logger now that we’ve the above middleware.</p><pre class=" language-diff"><code class="language-diff">  # src/logger.py  import json  import logging  from logging import Formatter  class JsonFormatter(Formatter):      def __init__(self):          super(JsonFormatter, self).__init__()      def format(self, record):          json_record = {}          json_record["message"] = record.getMessage()<span class="token inserted">+         if "req" in record.__dict__:</span><span class="token inserted">+             json_record["req"] = record.__dict__["req"]</span><span class="token inserted">+         if "res" in record.__dict__:</span><span class="token inserted">+             json_record["res"] = record.__dict__["res"]</span><span class="token inserted">+         if record.levelno == logging.ERROR and record.exc_info:</span><span class="token inserted">+             json_record["err"] = self.formatException(record.exc_info)</span>          return json.dumps(json_record)  logger = logging.root  handler = logging.StreamHandler()  handler.setFormatter(JsonFormatter())  logger.handlers = [handler]  logger.setLevel(logging.DEBUG)<span class="token inserted">+ logging.getLogger("uvicorn.access").disabled = True</span></code></pre><p>Finally, add the middleware to the application.</p><pre class=" language-diff"><code class="language-diff">  # src/main.py  import uvicorn  from fastapi import FastAPI  from src.logger import logger<span class="token inserted">+ from src.log_middleware import LogMiddleware</span>  app = FastAPI()<span class="token inserted">+ app.add_middleware(LogMiddleware)</span>  @app.get("/")  async def hello():      logger.info("Hello")      return "Hello World"  if __name__ == "__main__":      uvicorn.run(app, host="0.0.0.0", port=8000, log_config=None)</code></pre><p>You can now see structured JSON logs for the requests:</p><pre class=" language-json"><code class="language-json"><span class="token punctuation">{</span><span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"Hello"</span><span class="token punctuation">}</span><span class="token punctuation">{</span><span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"Incoming request"</span><span class="token punctuation">,</span> <span class="token property">"req"</span><span class="token operator">:</span> <span class="token punctuation">{</span><span class="token property">"method"</span><span class="token operator">:</span> <span class="token string">"GET"</span><span class="token punctuation">,</span> <span class="token property">"url"</span><span class="token operator">:</span> <span class="token string">"http://0.0.0.0:8000/"</span><span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token property">"res"</span><span class="token operator">:</span> <span class="token punctuation">{</span><span class="token property">"status_code"</span><span class="token operator">:</span> <span class="token number">200</span><span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><p>You can find the full codebase in this <a href="https://github.com/sheshbabu/fastapi-structured-json-logging-demo" target="_blank" rel="noopener">repo</a>.</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;Let’s start with a simple FastAPI example:&lt;/p&gt;
&lt;pre class=&quot; language-python&quot;&gt;&lt;code class=&quot;language-python&quot;&gt;&lt;span class=&quot;token comment&quot; sp
      
    
    </summary>
    
    
      <category term="Python" scheme="https://www.sheshbabu.com/tags/Python/"/>
    
      <category term="Logging" scheme="https://www.sheshbabu.com/tags/Logging/"/>
    
      <category term="FastAPI" scheme="https://www.sheshbabu.com/tags/FastAPI/"/>
    
  </entry>
  
  <entry>
    <title>Why use Azure OpenAI when you have OpenAI?</title>
    <link href="https://www.sheshbabu.com/posts/why-use-azure-openai-when-you-have-openai/"/>
    <id>https://www.sheshbabu.com/posts/why-use-azure-openai-when-you-have-openai/</id>
    <published>2023-06-21T17:03:26.000Z</published>
    <updated>2024-05-19T06:17:28.273Z</updated>
    
    <content type="html"><![CDATA[<p>In addition to the <a href="https://chat.openai.com/" target="_blank" rel="noopener">ChatGPT</a> application, OpenAI has also released APIs that allow you to integrate it in your products. Confusingly, Microsoft has also released something called <a href="https://azure.microsoft.com/en-ca/products/cognitive-services/openai-service" target="_blank" rel="noopener">Azure OpenAI Service</a>, which appears to be doing the same.</p><p>So, what’s the difference?</p><p><img src="/images/2023-why-use-azure-openai-when-you-have-openai/2023-why-use-azure-openai-when-you-have-openai.png" alt=""></p><h2 id="Data-Privacy"><a href="#Data-Privacy" class="headerlink" title="Data Privacy"></a>Data Privacy</h2><p>One of the biggest concerns people have with these LLM services is that they might use the company data provided by you to further train the model. This is problematic when you’re working with sensitive data from industries like healthcare, finance, etc. Both <a href="https://learn.microsoft.com/en-ca/azure/cognitive-services/openai/faq" target="_blank" rel="noopener">Azure OpenAI Service</a> and <a href="https://platform.openai.com/docs/models/how-we-use-your-data" target="_blank" rel="noopener">OpenAI</a> don’t use your data for training.</p><p>Another concern people have is whether their prompts or conversation history is being stored by these services. Both <a href="https://learn.microsoft.com/en-ca/legal/cognitive-services/openai/data-privacy#how-does-the-azure-openai-service-process-data" target="_blank" rel="noopener">Azure</a> and <a href="https://platform.openai.com/docs/models/how-we-use-your-data" target="_blank" rel="noopener">OpenAI</a> log your API requests and retain them for 30 days to identify abuse. However, you can opt-out of this from both services.</p><p>So, when it comes to data privacy, both Azure OpenAI Service and OpenAI are identical.</p><h2 id="Data-Residency"><a href="#Data-Residency" class="headerlink" title="Data Residency"></a>Data Residency</h2><p>When you’re working with cloud services, choosing the right region is very important. By selecting a region that’s closest to your customers, you can reduce the network latency and thereby provide better performance to your customers.</p><p>Being able to choose a region is also important for compliance reasons. If you work in regulated industries like healthcare, finance, etc, you might be required by local regulations to store your data in only a few geographical locations like Europe, US, etc.</p><p>Azure OpenAI Service is available in <a href="https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models#model-summary-table-and-region-availability" target="_blank" rel="noopener">multiple regions</a> like East US, South Central US, UK South, West Europe, France Central. OpenAI currently doesn’t support multiple regions.</p><h2 id="Infrastructure-Security"><a href="#Infrastructure-Security" class="headerlink" title="Infrastructure Security"></a>Infrastructure Security</h2><p>Defense in depth is an important concept in security. Azure provides network security tools such as <a href="https://learn.microsoft.com/en-us/azure/cognitive-services/cognitive-services-virtual-networks" target="_blank" rel="noopener">VNets</a> and Private Link which can be used to add more security layers like network isolation, routing traffic through private backbone instead of public internet, using firewalls to limit inbound traffic only to certain IP addresses etc which can be used with Azure OpenAI Service.</p><p>In addition to the network security, access control is also an important part of infrastructure security - making sure that only the correct people can access the cloud resources. Azure provides tools such as Azure AD and Azure RBAC for handling this.</p><p>In comparison, OpenAI doesn’t have such sophisticated security tools.</p><h2 id="Integration-with-Microsoft-ecosystem"><a href="#Integration-with-Microsoft-ecosystem" class="headerlink" title="Integration with Microsoft ecosystem"></a>Integration with Microsoft ecosystem</h2><p>This is one of the obvious benefits of using Azure OpenAI - close integration with other Microsoft/Azure products such as <a href="https://learn.microsoft.com/en-gb/azure/search/search-howto-index-sharepoint-online" target="_blank" rel="noopener">SharePoint</a>, <a href="https://learn.microsoft.com/en-us/ai-builder/azure-openai-model-pauto" target="_blank" rel="noopener">Power Automate</a>, <a href="https://learn.microsoft.com/en-gb/azure/search/search-howto-connecting-azure-sql-database-to-azure-search-using-indexers" target="_blank" rel="noopener">Azure SQL</a>, <a href="https://learn.microsoft.com/en-gb/azure/search/search-howto-indexing-azure-blob-storage" target="_blank" rel="noopener">Azure Blob Storage</a> etc.</p><p>As most enterprises use Microsoft products such as SharePoint, O365 etc, building internal applications that leverage LLMs would be easier to accomplish using Azure OpenAI Service. OpenAI doesn’t have this level of integration.</p><h2 id="Model-parity"><a href="#Model-parity" class="headerlink" title="Model parity"></a>Model parity</h2><p>OpenAI <a href="https://openai.com/blog/function-calling-and-other-api-updates" target="_blank" rel="noopener">recently released</a> new versions of GPT-4 and GPT-3.5 Turbo. The GPT-4 model (0613) introduced the function calling feature. GPT-3.5 Turbo also received an update (0613) which included function calling, a new 16k context variant, and cost reduction.</p><p>These changes have not propagated to <a href="https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models#model-summary-table-and-region-availability" target="_blank" rel="noopener">Azure OpenAI</a> yet.</p><h2 id="Pricing"><a href="#Pricing" class="headerlink" title="Pricing"></a>Pricing</h2><p>We pay based on the type of the model, its max token limit and how many tokens we consume. Surprisingly, both <a href="https://openai.com/pricing" target="_blank" rel="noopener">OpenAI</a> and <a href="https://azure.microsoft.com/en-gb/pricing/details/cognitive-services/openai-service/" target="_blank" rel="noopener">Azure</a> have the same pricing model!</p><table><thead><tr><th>Service</th><th>Model</th><th>Context</th><th>Input</th><th>Output</th></tr></thead><tbody><tr><td>OpenAI</td><td>GPT-4</td><td>8K</td><td>$0.03</td><td>$0.06</td></tr><tr><td>Azure OpenAI</td><td>GPT-4</td><td>8K</td><td>$0.03</td><td>$0.06</td></tr><tr><td>OpenAI</td><td>GPT-4</td><td>32K</td><td>$0.06</td><td>$0.12</td></tr><tr><td>Azure OpenAI</td><td>GPT-4</td><td>32K</td><td>$0.06</td><td>$0.12</td></tr><tr><td>OpenAI</td><td>GPT-3.5 Turbo</td><td>4K</td><td>$0.0015</td><td>$0.002</td></tr><tr><td>Azure OpenAI</td><td>GPT-3.5 Turbo</td><td>4K</td><td>$0.002</td><td>$0.002</td></tr><tr><td>OpenAI</td><td>GPT-3.5 Turbo</td><td>16K</td><td>$0.003</td><td>$0.004</td></tr><tr><td>Azure OpenAI</td><td>GPT-3.5 Turbo</td><td>16K</td><td>N/A</td><td>N/A</td></tr></tbody></table><p>The price is per 1000 tokens.</p><h2 id="Rate-Limits"><a href="#Rate-Limits" class="headerlink" title="Rate Limits"></a>Rate Limits</h2><p>The API requests are rate limited in two ways - either by requests per minute (RPM) which might be familiar to you working with other APIs, and also tokens per minute (TPM). You hit these limits either by sending too many requests (RPM limit) or by sending bigger requests (TPM limit).</p><p>Both offer different rate limits but of the same order of magnitude. In <a href="https://learn.microsoft.com/en-ca/azure/cognitive-services/openai/quotas-limits" target="_blank" rel="noopener">Azure</a>, the limits are <a href="https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/quota" target="_blank" rel="noopener">scoped to a region</a>, so you can spin up new regions to get around this issue. <a href="https://platform.openai.com/docs/guides/rate-limits" target="_blank" rel="noopener">OpenAI</a> doesn’t have multiple region support yet.</p><table><thead><tr><th>Service</th><th>Model</th><th>Context</th><th>TPM (In thousands)</th><th>RPM</th></tr></thead><tbody><tr><td>OpenAI</td><td>GPT-4</td><td>8K</td><td>40</td><td>200</td></tr><tr><td>Azure OpenAI</td><td>GPT-4</td><td>8K</td><td>20</td><td>120</td></tr><tr><td>OpenAI</td><td>GPT-4</td><td>32K</td><td>150</td><td>20</td></tr><tr><td>Azure OpenAI</td><td>GPT-4</td><td>32K</td><td>60</td><td>360</td></tr><tr><td>OpenAI</td><td>GPT-3.5 Turbo</td><td>4K</td><td>90</td><td>3500</td></tr><tr><td>Azure OpenAI</td><td>GPT-3.5 Turbo</td><td>4K</td><td>240</td><td>1440</td></tr><tr><td>OpenAI</td><td>GPT-3.5 Turbo</td><td>16K</td><td>180</td><td>3500</td></tr><tr><td>Azure OpenAI</td><td>GPT-3.5 Turbo</td><td>16K</td><td>N/A</td><td>N/A</td></tr></tbody></table><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>In summary, choose Azure OpenAI if you have compliance and security requirements, and if your organization is already using Microsoft’s products or Azure infrastructure.</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;In addition to the &lt;a href=&quot;https://chat.openai.com/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;ChatGPT&lt;/a&gt; application, OpenAI has also released AP
      
    
    </summary>
    
    
      <category term="LLM" scheme="https://www.sheshbabu.com/tags/LLM/"/>
    
  </entry>
  
  <entry>
    <title>A Gentle Introduction to LLMs for Product Teams</title>
    <link href="https://www.sheshbabu.com/posts/gentle-introduction-to-llms/"/>
    <id>https://www.sheshbabu.com/posts/gentle-introduction-to-llms/</id>
    <published>2023-06-12T17:39:29.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<p>There has been explosive growth in Generative AI and LLMs (Large Language Models) in the past few months. Not only we are seeing the launch of new products like <a href="https://chat.openai.com/" target="_blank" rel="noopener">ChatGPT</a>, <a href="https://www.midjourney.com/" target="_blank" rel="noopener">Midjourney</a> etc, we also see established products like <a href="https://www.notion.so/product/ai" target="_blank" rel="noopener">Notion</a>, <a href="https://github.com/features/copilot" target="_blank" rel="noopener">GitHub</a>, etc incorporating LLMs to provide new features to users.</p><p>If you’re working in a product team as a designer, product manager or a non-ML developer, you might be wondering if you can use LLMs in your products. But it might have been hard to understand the terminologies like “memory”, “tokens”, “langchain” etc or know what the capabilities and limitations of these LLMs are.</p><p>In this post, I’ll try to explain LLMs in an easy to understand manner.</p><h2 id="Prompts-and-Prompt-Engineering"><a href="#Prompts-and-Prompt-Engineering" class="headerlink" title="Prompts and Prompt Engineering"></a>Prompts and Prompt Engineering</h2><p>Let’s start with something most of us are familiar with - ChatGPT. We ask it a question and it gives an answer. We ask a followup question and it gives another answer. The questions we ask the LLMs are called “prompts”.</p><p><img src="/images/2023-gentle-introduction-to-llms/2023-gentle-introduction-to-llms-01.png" alt=""></p><p>You might have noticed that you can also ask it to change the way it answers. You can ask it to make the answer concise, change the tone to be professional, etc. You can also use more advanced techniques like <a href="https://learnprompting.org/docs/intermediate/zero_shot_cot" target="_blank" rel="noopener">using examples to getter better answers</a>, <a href="https://learnprompting.org/docs/intermediate/chain_of_thought" target="_blank" rel="noopener">asking the model to do step-by-step reasoning</a>, reduce hallucinations etc. These methods of manipulating the LLM’s output by wording your prompts differently or adding instructions is called “prompt engineering”.</p><p><img src="/images/2023-gentle-introduction-to-llms/2023-gentle-introduction-to-llms-02.png" alt=""></p><h2 id="Token-Limits"><a href="#Token-Limits" class="headerlink" title="Token Limits"></a>Token Limits</h2><p>When we build applications, we usually add restrictions on the user inputs. For example, we might make the username field to be max 100 characters, or the product review written by the user to have max 1000 characters etc.</p><p>LLMs have similar restrictions, but instead of measuring in number of characters, they measure in number of “tokens”. To put it simply, a “token” is either a word or partial word. Here’s an example of how the sentence “Premature optimization is the root of all evil.” is split into tokens using the <a href="https://platform.openai.com/tokenizer" target="_blank" rel="noopener">OpenAI Tokenizer</a> tool:</p><p><img src="/images/2023-gentle-introduction-to-llms/2023-gentle-introduction-to-llms-03.png" alt=""></p><p>The sentence contains 8 words but it’s split into 10 tokens. Notice how the first word is split into two tokens, and how the period at the end is its own token.</p><p>A good rule of thumb is to think of a token as 3/4th of a word. So 75 words are 100 tokens.</p><p>Some models like <code>gpt-3.5-turbo</code> have 4096 max token limit (~ 3000 words), some like <code>gpt-4-32k</code> have 32,768 (~ 25,000 words). And this token limit affects both the input and output of the model. If the input takes up most of the token limit, there’s less number of tokens available for the model to output.</p><h2 id="Memory-and-Stateless-Models"><a href="#Memory-and-Stateless-Models" class="headerlink" title="Memory and Stateless Models"></a>Memory and Stateless Models</h2><p>One thing we don’t notice about ChatGPT is how it remembers what we tell it. We don’t notice this as it’s an intuitive behavior - if we’re chatting with our friend, we expect them to remember what we told them before. Notice how it remembers my name:</p><p><img src="/images/2023-gentle-introduction-to-llms/2023-gentle-introduction-to-llms-04.png" alt=""></p><p>But LLMs are actually stateless - they have no memory of previous conversation! Then how do they “remember” what we said before? It’s because ChatGPT passes the previous conversation history along with the latest question. As you can imagine, this approach only works to some extent until you start hitting the token limits.</p><h2 id="Consequence-of-Stateless-Models-and-Token-Limits"><a href="#Consequence-of-Stateless-Models-and-Token-Limits" class="headerlink" title="Consequence of Stateless Models and Token Limits"></a>Consequence of Stateless Models and Token Limits</h2><p>At the risk of oversimplifying, most applications we build basically have a database and a frontend for the users to update/query the database. If you’re working on such a product, your immediate instinct is to see if you can replace/augment your frontend with a chat interface so your users can update/query your database by “chatting” with an LLM.</p><p>But how does the LLM know the structure of your database - what columns it has, the format of the data inside them, etc? Should we just dump the contents of the database into a prompt and ask it follow up questions? From what we know of token limits, this is not possible unless your database is tiny. You also can’t ask it to memorize the database in one prompt and ask followup questions.</p><p>People have come up with creative workarounds for this problem:</p><ul><li>Instead of dumping the entire database into the prompt, we can include our database schema and a few example records so it can generate a SQL query for us to query the database on its behalf.</li><li>Same goes for documents - instead of feeding it documents, we can search the documents based on the user’s prompt, retrieve the relevant passage from the document, add it to the user prompt and finally ask the LLM to reformat the document passage based on the user’s prompt.</li></ul><p>These techniques are known as “Retrieval-Augmented Generation”.</p><h2 id="Data-Privacy"><a href="#Data-Privacy" class="headerlink" title="Data Privacy"></a>Data Privacy</h2><p>If your product deals with sensitive information like PII, health records etc, more care should be taken while using LLMs. The data passed in prompts can be leaked if not stored properly or it can be used for training the model which could be leaked to other users when they ask similar questions.</p><h2 id="Hallucination"><a href="#Hallucination" class="headerlink" title="Hallucination"></a>Hallucination</h2><p>Sometimes, LLMs can give confidently wrong answers. While this might be acceptable in some scenarios, this severely limits their usefulness when you want to integrate them into your product.</p><h2 id="LangChain"><a href="#LangChain" class="headerlink" title="LangChain"></a>LangChain</h2><p>It’s a library for building applications powered by LLMs. You can think of it similar to using <a href="https://react.dev" target="_blank" rel="noopener">React</a> for UI or <a href="https://rubyonrails.org" target="_blank" rel="noopener">Rails</a> for Backend apps etc. It simplifies the development of LLM apps by providing useful features like prompt templates, memory, agents etc and allow us to compose (chain) these features together to build a workflow.</p><h2 id="Agents"><a href="#Agents" class="headerlink" title="Agents"></a>Agents</h2><p>LLMs are good with reasoning and language skills. We can ask a question, if the details provided are insufficient, the LLM asks a clarifying question, we can answer its questions to get the final answer:</p><p><img src="/images/2023-gentle-introduction-to-llms/2023-gentle-introduction-to-llms-05.png" alt=""></p><p>What if we remove ourselves from the above flow? What if it can use “tools” like search Wikipedia, make API calls, or run a Python script? What if we can combine both the reasoning and the ability to use “tools”? This is called an “agent”.</p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>Understanding the capabilities and limitations of LLMs can help us better evaluate how to integrate it into our products. It’s important to note that this is a fast moving space, so some of these limitations might be solved at the model level or new products could come up to mitigate the concerns. It’s helpful to keep up with these advancements so we can make the right decisions and tradeoffs.</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;There has been explosive growth in Generative AI and LLMs (Large Language Models) in the past few months. Not only we are seeing the laun
      
    
    </summary>
    
    
      <category term="LLM" scheme="https://www.sheshbabu.com/tags/LLM/"/>
    
  </entry>
  
  <entry>
    <title>Detecting Clusters in Graphs using NetworkX</title>
    <link href="https://www.sheshbabu.com/posts/detecting-clusters-in-graphs-using-networkx/"/>
    <id>https://www.sheshbabu.com/posts/detecting-clusters-in-graphs-using-networkx/</id>
    <published>2023-03-12T13:43:07.000Z</published>
    <updated>2024-05-19T06:17:28.270Z</updated>
    
    <content type="html"><![CDATA[<p>Clusters of all connected nodes inside a graph is commonly known as “connected components”</p><p><img src="/images/2023-detecting-clusters-in-graphs-using-networkx/graph-clusters-connected-components.png" alt=""></p><p>Let’s recreate the above graph in NetworkX:</p><pre class=" language-python"><code class="language-python"><span class="token keyword">import</span> networkx <span class="token keyword">as</span> nxG <span class="token operator">=</span> nx<span class="token punctuation">.</span>Graph<span class="token punctuation">(</span><span class="token punctuation">)</span>G<span class="token punctuation">.</span>add_nodes_from<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">"A"</span><span class="token punctuation">,</span> <span class="token string">"B"</span><span class="token punctuation">,</span> <span class="token string">"C"</span><span class="token punctuation">,</span> <span class="token string">"D"</span><span class="token punctuation">,</span> <span class="token string">"E"</span><span class="token punctuation">,</span> <span class="token string">"F"</span><span class="token punctuation">,</span> <span class="token string">"G"</span><span class="token punctuation">,</span> <span class="token string">"H"</span><span class="token punctuation">,</span> <span class="token string">"I"</span><span class="token punctuation">,</span> <span class="token string">"J"</span><span class="token punctuation">,</span> <span class="token string">"K"</span><span class="token punctuation">,</span> <span class="token string">"L"</span><span class="token punctuation">,</span> <span class="token string">"M"</span><span class="token punctuation">,</span> <span class="token string">"N"</span><span class="token punctuation">]</span><span class="token punctuation">)</span>G<span class="token punctuation">.</span>add_edges_from<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token punctuation">(</span><span class="token string">"B"</span><span class="token punctuation">,</span> <span class="token string">"A"</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">"E"</span><span class="token punctuation">,</span> <span class="token string">"F"</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">"E"</span><span class="token punctuation">,</span> <span class="token string">"D"</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">"G"</span><span class="token punctuation">,</span> <span class="token string">"H"</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">"I"</span><span class="token punctuation">,</span> <span class="token string">"H"</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">"J"</span><span class="token punctuation">,</span> <span class="token string">"I"</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">"J"</span><span class="token punctuation">,</span> <span class="token string">"K"</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">"M"</span><span class="token punctuation">,</span> <span class="token string">"L"</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre><p>We then call the <a href="https://networkx.org/documentation/stable/reference/algorithms/generated/networkx.algorithms.components.connected_components.html" target="_blank" rel="noopener">connected_components</a> function of NetworkX.</p><pre class=" language-python"><code class="language-python">list<span class="token punctuation">(</span>nx<span class="token punctuation">.</span>connected_components<span class="token punctuation">(</span>G<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre><p>Which will give the list of all clusters or connected components of this graph:</p><pre class=" language-python"><code class="language-python"><span class="token punctuation">[</span><span class="token punctuation">{</span><span class="token string">'A'</span><span class="token punctuation">,</span> <span class="token string">'B'</span><span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span><span class="token string">'C'</span><span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span><span class="token string">'D'</span><span class="token punctuation">,</span> <span class="token string">'E'</span><span class="token punctuation">,</span> <span class="token string">'F'</span><span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span><span class="token string">'G'</span><span class="token punctuation">,</span> <span class="token string">'H'</span><span class="token punctuation">,</span> <span class="token string">'I'</span><span class="token punctuation">,</span> <span class="token string">'J'</span><span class="token punctuation">,</span> <span class="token string">'K'</span><span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span><span class="token string">'L'</span><span class="token punctuation">,</span> <span class="token string">'M'</span><span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span><span class="token string">'N'</span><span class="token punctuation">}</span><span class="token punctuation">]</span></code></pre>]]></content>
    
    <summary type="html">
    
      Clusters of all connected nodes inside a graph is commonly known as “connected components”
    
    </summary>
    
    
      <category term="Python" scheme="https://www.sheshbabu.com/tags/Python/"/>
    
      <category term="Entity Resolution" scheme="https://www.sheshbabu.com/tags/Entity-Resolution/"/>
    
      <category term="Graph" scheme="https://www.sheshbabu.com/tags/Graph/"/>
    
      <category term="NetworkX" scheme="https://www.sheshbabu.com/tags/NetworkX/"/>
    
  </entry>
  
  <entry>
    <title>Generating Unique Pairs in Python</title>
    <link href="https://www.sheshbabu.com/posts/generating-unique-pairs-in-python/"/>
    <id>https://www.sheshbabu.com/posts/generating-unique-pairs-in-python/</id>
    <published>2023-03-12T13:33:59.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<p>Let’s say we’ve a list of entities, this can be anything like people names, company names, etc from which we’d like to generate unique pairs</p><pre class=" language-python"><code class="language-python">entities <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">"A"</span><span class="token punctuation">,</span> <span class="token string">"B"</span><span class="token punctuation">,</span> <span class="token string">"C"</span><span class="token punctuation">,</span> <span class="token string">"D"</span><span class="token punctuation">,</span> <span class="token string">"E"</span><span class="token punctuation">]</span></code></pre><p>We can use the <code>distinct_combinations</code> function from <a href="https://more-itertools.readthedocs.io/en/stable/index.html" target="_blank" rel="noopener">more_itertools</a> package</p><pre class=" language-python"><code class="language-python"><span class="token keyword">from</span> more_itertools <span class="token keyword">import</span> distinct_combinationscandidate_pairs <span class="token operator">=</span> distinct_combinations<span class="token punctuation">(</span>entities<span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">)</span></code></pre><p>This will generate all the possible pairs with duplicates removed:</p><pre class=" language-python"><code class="language-python"><span class="token punctuation">[</span><span class="token punctuation">(</span><span class="token string">'A'</span><span class="token punctuation">,</span> <span class="token string">'B'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">'A'</span><span class="token punctuation">,</span> <span class="token string">'C'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">'A'</span><span class="token punctuation">,</span> <span class="token string">'D'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">'A'</span><span class="token punctuation">,</span> <span class="token string">'E'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">'B'</span><span class="token punctuation">,</span> <span class="token string">'C'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">'B'</span><span class="token punctuation">,</span> <span class="token string">'D'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">'B'</span><span class="token punctuation">,</span> <span class="token string">'E'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">'C'</span><span class="token punctuation">,</span> <span class="token string">'D'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">'C'</span><span class="token punctuation">,</span> <span class="token string">'E'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">'D'</span><span class="token punctuation">,</span> <span class="token string">'E'</span><span class="token punctuation">)</span><span class="token punctuation">]</span></code></pre>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;Let’s say we’ve a list of entities, this can be anything like people names, company names, etc from which we’d like to generate unique pa
      
    
    </summary>
    
    
      <category term="Python" scheme="https://www.sheshbabu.com/tags/Python/"/>
    
      <category term="Entity Resolution" scheme="https://www.sheshbabu.com/tags/Entity-Resolution/"/>
    
  </entry>
  
  <entry>
    <title>Loading Amplitude Events Data into Pandas DataFrame</title>
    <link href="https://www.sheshbabu.com/posts/loading-amplitude-events-data-into-pandas-dataframe/"/>
    <id>https://www.sheshbabu.com/posts/loading-amplitude-events-data-into-pandas-dataframe/</id>
    <published>2023-03-12T13:20:01.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<p>Amplitude allows you to export the events data from their platform using their <a href="https://www.docs.developers.amplitude.com/analytics/apis/export-api/" target="_blank" rel="noopener">Export API</a>. Let’s see how we can get that exported data loaded into a Pandas dataframe.</p><p>First, get the Amplitude API credentials for your project.</p><pre class=" language-python"><code class="language-python">API_KEY <span class="token operator">=</span> <span class="token string">"xxxx"</span>SECRET_KEY <span class="token operator">=</span> <span class="token string">"yyyy"</span></code></pre><p>The data is exported based on a date range, but there are limitations on the size of the data which can be exported in a single API call. So depending on the size of the data, we might need to make multiple API calls.</p><p>For this post, I’m assuming a day’s worth of data falls within this limit. Let’s export the data for the past 7 days.</p><pre class=" language-python"><code class="language-python"><span class="token keyword">from</span> datetime <span class="token keyword">import</span> date<span class="token punctuation">,</span> timedeltaNUM_DAYS <span class="token operator">=</span> <span class="token number">7</span>end_date <span class="token operator">=</span> date<span class="token punctuation">.</span>today<span class="token punctuation">(</span><span class="token punctuation">)</span>start_date <span class="token operator">=</span> end_date <span class="token operator">-</span> timedelta<span class="token punctuation">(</span>days<span class="token operator">=</span>NUM_DAYS<span class="token punctuation">)</span></code></pre><p>Since the API response is a zip file, let’s create a temp directory to store them.</p><pre class=" language-python"><code class="language-python"><span class="token keyword">from</span> pathlib <span class="token keyword">import</span> Pathraw_files_path <span class="token operator">=</span> Path<span class="token punctuation">.</span>cwd<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>joinpath<span class="token punctuation">(</span><span class="token string">"./temp/amplitude"</span><span class="token punctuation">)</span>Path<span class="token punctuation">.</span>mkdir<span class="token punctuation">(</span>raw_files_path<span class="token punctuation">,</span> parents<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">,</span> exist_ok<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span></code></pre><p>As mentioned before, we’re splitting the specified date range into multiple days, and making an API request for each day. The date range for the API request is specified using the <code>start</code> and <code>end</code> query parameters formatted in the <code>YYYYMMDDTHH</code> format. We use the <code>API_KEY</code> and <code>SECRET_KEY</code> from before, encode them and send them along as the <code>Authorization</code> header. The response is then written to a ZIP file in the temp directory.</p><pre class=" language-python"><code class="language-python"><span class="token keyword">import</span> requests<span class="token keyword">from</span> base64 <span class="token keyword">import</span> b64encode<span class="token keyword">from</span> datetime <span class="token keyword">import</span> timedeltaurl <span class="token operator">=</span> <span class="token string">"https://amplitude.com/api/2/export"</span>token <span class="token operator">=</span> b64encode<span class="token punctuation">(</span>f<span class="token string">"{API_KEY}:{SECRET_KEY}"</span><span class="token punctuation">.</span>encode<span class="token punctuation">(</span><span class="token string">"utf-8"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">.</span>decode<span class="token punctuation">(</span><span class="token string">"utf-8"</span><span class="token punctuation">)</span>headers <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">"Authorization"</span><span class="token punctuation">:</span> f<span class="token string">"Basic {token}"</span><span class="token punctuation">}</span><span class="token keyword">for</span> idx <span class="token keyword">in</span> range<span class="token punctuation">(</span><span class="token punctuation">(</span>end_date <span class="token operator">-</span> start_date<span class="token punctuation">)</span><span class="token punctuation">.</span>days<span class="token punctuation">)</span><span class="token punctuation">:</span>    current_date <span class="token operator">=</span> end_date <span class="token operator">-</span> timedelta<span class="token punctuation">(</span>days<span class="token operator">=</span>idx<span class="token punctuation">)</span>    current_date <span class="token operator">=</span> current_date<span class="token punctuation">.</span>strftime<span class="token punctuation">(</span><span class="token string">"%Y%m%d"</span><span class="token punctuation">)</span>    params <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">"start"</span><span class="token punctuation">:</span> f<span class="token string">"{current_date}T00"</span><span class="token punctuation">,</span> <span class="token string">"end"</span><span class="token punctuation">:</span> f<span class="token string">"{current_date}T23"</span><span class="token punctuation">}</span>    res <span class="token operator">=</span> requests<span class="token punctuation">.</span>get<span class="token punctuation">(</span>url<span class="token punctuation">,</span> params<span class="token operator">=</span>params<span class="token punctuation">,</span> headers<span class="token operator">=</span>headers<span class="token punctuation">)</span>    open<span class="token punctuation">(</span>raw_files_path<span class="token punctuation">.</span>joinpath<span class="token punctuation">(</span>f<span class="token string">"{current_date}.zip"</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">"wb"</span><span class="token punctuation">)</span><span class="token punctuation">.</span>write<span class="token punctuation">(</span>res<span class="token punctuation">.</span>content<span class="token punctuation">)</span></code></pre><p>Once we’re done downloading the ZIP files, we iterate through them and extract the contents.</p><pre class=" language-python"><code class="language-python"><span class="token keyword">import</span> shutil<span class="token keyword">for</span> path <span class="token keyword">in</span> list<span class="token punctuation">(</span>raw_files_path<span class="token punctuation">.</span>iterdir<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>    <span class="token keyword">if</span> path<span class="token punctuation">.</span>suffix <span class="token operator">!=</span> <span class="token string">".zip"</span><span class="token punctuation">:</span>        <span class="token keyword">continue</span>    shutil<span class="token punctuation">.</span>unpack_archive<span class="token punctuation">(</span>path<span class="token punctuation">,</span> raw_files_path<span class="token punctuation">)</span></code></pre><p>Finally, we iterate through the compressed JSON files and load them into dataframe <code>df</code>.</p><pre class=" language-python"><code class="language-python"><span class="token keyword">import</span> pandas <span class="token keyword">as</span> pddf <span class="token operator">=</span> None<span class="token keyword">for</span> path <span class="token keyword">in</span> list<span class="token punctuation">(</span>raw_files_path<span class="token punctuation">.</span>iterdir<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>    <span class="token keyword">if</span> <span class="token operator">not</span> path<span class="token punctuation">.</span>is_dir<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>        <span class="token keyword">continue</span>    <span class="token keyword">for</span> file_path <span class="token keyword">in</span> list<span class="token punctuation">(</span>path<span class="token punctuation">.</span>iterdir<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>        <span class="token keyword">if</span> <span class="token operator">not</span> file_path<span class="token punctuation">.</span>name<span class="token punctuation">.</span>endswith<span class="token punctuation">(</span><span class="token string">".json.gz"</span><span class="token punctuation">)</span><span class="token punctuation">:</span>            <span class="token keyword">continue</span>        <span class="token keyword">if</span> df <span class="token keyword">is</span> None<span class="token punctuation">:</span>            df <span class="token operator">=</span> pd<span class="token punctuation">.</span>read_json<span class="token punctuation">(</span>file_path<span class="token punctuation">,</span> lines<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span>        <span class="token keyword">else</span><span class="token punctuation">:</span>            new_df <span class="token operator">=</span> pd<span class="token punctuation">.</span>read_json<span class="token punctuation">(</span>file_path<span class="token punctuation">,</span> lines<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span>            df <span class="token operator">=</span> pd<span class="token punctuation">.</span>concat<span class="token punctuation">(</span><span class="token punctuation">[</span>df<span class="token punctuation">,</span> new_df<span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre>]]></content>
    
    <summary type="html">
    
      Amplitude allows you to export the events data from their platform using their Export API. Let’s see how we can get that exported data loaded into a Pandas dataframe.
    
    </summary>
    
    
      <category term="Python" scheme="https://www.sheshbabu.com/tags/Python/"/>
    
      <category term="Analytics" scheme="https://www.sheshbabu.com/tags/Analytics/"/>
    
      <category term="Pandas" scheme="https://www.sheshbabu.com/tags/Pandas/"/>
    
      <category term="Amplitude" scheme="https://www.sheshbabu.com/tags/Amplitude/"/>
    
  </entry>
  
  <entry>
    <title>Make your backend more reliable using Nginx caching proxy</title>
    <link href="https://www.sheshbabu.com/posts/nginx-caching-proxy/"/>
    <id>https://www.sheshbabu.com/posts/nginx-caching-proxy/</id>
    <published>2022-07-10T04:01:04.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<p>Most of us are familiar with Nginx - it’s a very popular Web Server and Reverse Proxy. But do you know that you can also use it as a Caching Proxy?</p><p>Now, you’re probably wondering why would anyone want to do something like that - can’t you just update your service to cache data in Redis or Memcached? What are the upsides of externalizing the cache to a separate layer outside your service?</p><p>These are some scenarios where this might be useful:</p><ul><li>You want to serve cached data even when your service goes down</li><li>You want to serve cached data when your service takes too long to respond</li><li>You want to protect your service when there’s a lot of load</li><li>You have a legacy system that you want to be made more reliable and performant but you can’t change the code</li><li>You want to make a external 3rd party service more reliable and performant</li><li>You’re using a polyglot microservices architecture and want a standard way of caching requests</li></ul><p>Now that you’re interested, let’s go through the implementation one step at a time.</p><h2 id="Simple-proxy"><a href="#Simple-proxy" class="headerlink" title="Simple proxy"></a>Simple proxy</h2><p>Let’s start off with a simple implementation. You have a Nginx server that just proxies all the calls to your backend service without any caching.</p><p><img src="/images/2022-nginx-caching-proxy/2022-nginx-caching-proxy-2-simple-proxy.png" alt=""></p><p>This is what the Nginx config would look like:</p><pre class=" language-diff"><code class="language-diff">events {    worker_connections 1024;}http {    server {        listen 3000;        location / {            proxy_set_header Host $host;            proxy_pass http://my-backend-service/;        }    }}</code></pre><p>This runs the Nginx server in port <code>3000</code> and proxies all requests to <code>http://my-backend-service/</code></p><p>Read more: <a href="http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass" target="_blank" rel="noopener">proxy_pass</a>, <a href="http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header" target="_blank" rel="noopener">proxy_set_header</a></p><h2 id="Simple-cache-proxy"><a href="#Simple-cache-proxy" class="headerlink" title="Simple cache proxy"></a>Simple cache proxy</h2><p>Now let’s extend the above example with caching.</p><pre class=" language-diff"><code class="language-diff">events {    worker_connections 1024;}http {<span class="token inserted">+   proxy_cache_path /var/cache/nginx keys_zone=my_cache:10m;</span>    server {        listen 3000;<span class="token inserted">+       proxy_cache my_cache;</span>        location / {            proxy_set_header Host $host;            proxy_pass http://my-backend-service/;<span class="token inserted">+           proxy_cache_key $scheme://$host$uri$is_args$query_string;</span><span class="token inserted">+           proxy_cache_valid 200 10m;</span>        }    }}</code></pre><p>Some new directives are added, let’s go through them one by one. Nginx caches the responses in the disk, <code>proxy_cache_path</code> specifies the path where the responses are to be stored. <code>proxy_cache</code> defines the shared memory zone used for storing the cache keys and other metadata. <code>proxy_cache_key</code> defines the caching key. <code>proxy_cache_valid</code> specifies cache expiry, which can also be configured dynamically by sending <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control" target="_blank" rel="noopener">cache headers</a> from your backend service. Once the cache expires, the responses are no longer considered “fresh” and become “stale”.</p><p><img src="/images/2022-nginx-caching-proxy/2022-nginx-caching-proxy-3-simple-cache-proxy.png" alt=""></p><p>Nginx doesn’t immediately delete the stale responses and keeps them in disk. The reason why it doesn’t delete them would become more apparent in next few sections. We can use the parameter called <code>inactive</code> in the <code>proxy_cache_path</code> directive to control how long to keep the stale responses in disk before deleting them. It might be initially confusing how the <code>proxy_cache_valid</code> and <code>inactive</code> configuration work together as they seem to be doing similar things, let’s use the following example to understand this clearly:</p><p>If the <code>proxy_cache_valid</code> is set to 5m and <code>inactive</code> is set to 10m. If the first request comes at time T0 minutes and next request comes at T6 minutes, the second request would need to fetched from the backend service even though the data would still be in the disk as the cache has expired. If instead, the <code>proxy_cache_valid</code> is set to 10m and <code>inactive</code> is set to 5m, and the first request comes at time T0 minutes and next request comes at T6 minutes, even though the cache has not expired, the data is deleted from the disk so it needs to be fetched from backend service again.</p><p>Read more: <a href="http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_path" target="_blank" rel="noopener">proxy_cache_path</a>, <a href="http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache" target="_blank" rel="noopener">proxy_cache</a>, <a href="http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_key" target="_blank" rel="noopener">proxy_cache_key</a>, <a href="http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_valid" target="_blank" rel="noopener">proxy_cache_valid</a></p><h2 id="Temporarily-bypass-the-cache"><a href="#Temporarily-bypass-the-cache" class="headerlink" title="Temporarily bypass the cache"></a>Temporarily bypass the cache</h2><p>Sometimes you would need to temporarily bypass the cache and directly hit the backend service.</p><p><img src="/images/2022-nginx-caching-proxy/2022-nginx-caching-proxy-4-bypass-cache.png" alt=""></p><pre class=" language-diff"><code class="language-diff">events {    worker_connections 1024;}http {    proxy_cache_path /var/cache/nginx keys_zone=my_cache:10m;    server {        listen 3000;        proxy_cache my_cache;        location / {            proxy_set_header Host $host;            proxy_pass http://my-backend-service/;            proxy_cache_key $scheme://$host$uri$is_args$query_string;            proxy_cache_valid 200 10m;<span class="token inserted">+           proxy_cache_bypass $arg_should_bypass_cache;</span>        }    }}</code></pre><p>Here, we’ve added the <code>proxy_cache_bypass</code> directive with <code>$arg_should_bypass_cache</code> variable. If the request is sent with <code>?should_bypass_cache=true</code> query param, then the cache would be bypassed. You can also use cookies or HTTP headers instead of query params.</p><p>Read more: <a href="http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_bypass" target="_blank" rel="noopener">proxy_cache_bypass</a></p><h2 id="Serve-cached-data-when-backend-is-down-or-slow"><a href="#Serve-cached-data-when-backend-is-down-or-slow" class="headerlink" title="Serve cached data when backend is down or slow"></a>Serve cached data when backend is down or slow</h2><p>This is the killer application for using a separate caching proxy. When the backend service is down or takes too long to respond, we can configure Nginx to serve stale responses instead.</p><p><img src="/images/2022-nginx-caching-proxy/2022-nginx-caching-proxy-5-backend-down.png" alt=""></p><pre class=" language-diff"><code class="language-diff">events {    worker_connections 1024;}http {<span class="token deleted">-   proxy_cache_path /var/cache/nginx keys_zone=my_cache:10m;</span><span class="token inserted">+   proxy_cache_path /var/cache/nginx keys_zone=my_cache:10m inactive=1w;</span>    server {        listen 3000;        proxy_cache my_cache;        location / {            proxy_set_header Host $host;            proxy_pass http://my-backend-service/;            proxy_cache_key $scheme://$host$uri$is_args$query_string;            proxy_cache_valid 200 10m;            proxy_cache_bypass $arg_should_bypass_cache;<span class="token inserted">+           proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504 http_429;</span>        }    }}</code></pre><p>We’ve updated <code>proxy_cache_path</code> with the <code>inactive</code> parameter mentioned before and have added <code>proxy_cache_use_stale</code> directive with the following params:</p><p><code>error</code> - use stale if backend can’t be reached</p><p><code>timeout</code> - use stale if backend takes too long to respond</p><p><code>http_xxx</code> - use stale if backend returns these status codes</p><p>The timeout is set to <code>60s</code> by default and can be configured using <code>proxy_connect_timeout</code> directive.</p><p>Read more: <a href="http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_use_stale" target="_blank" rel="noopener">proxy_cache_use_stale</a>, <a href="http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream" target="_blank" rel="noopener">proxy_next_upstream</a>, <a href="http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout" target="_blank" rel="noopener">proxy_connect_timeout</a></p><h2 id="Prevent-cache-stampede"><a href="#Prevent-cache-stampede" class="headerlink" title="Prevent cache stampede"></a>Prevent cache stampede</h2><p>If you have an endpoint that’s slow and computationally expensive, and if there are many concurrent requests to that endpoint, the cache won’t be very helpful in reducing load to your backend service. This is called as “cache stampede” and in order to prevent this, we can use the <code>proxy_cache_lock</code> directive.</p><p><img src="/images/2022-nginx-caching-proxy/2022-nginx-caching-proxy-6-cache-stampede-1.png" alt=""></p><pre class=" language-diff"><code class="language-diff">events {    worker_connections 1024;}http {    proxy_cache_path /var/cache/nginx keys_zone=my_cache:10m inactive=1w;    server {        listen 3000;        proxy_cache my_cache;        location / {            proxy_set_header Host $host;            proxy_pass http://my-backend-service/;            proxy_cache_key $scheme://$host$uri$is_args$query_string;            proxy_cache_valid 200 10m;            proxy_cache_bypass $arg_should_bypass_cache;            proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504 http_429;<span class="token inserted">+           proxy_cache_lock on;</span>        }    }}</code></pre><p>This will ensure that only one request goes to your backend service and others wait for it to come back. One downside of this directive is that it adds extra <code>500ms</code> latency to the requests that have been waiting.</p><p><img src="/images/2022-nginx-caching-proxy/2022-nginx-caching-proxy-7-cache-stampede-2.png" alt=""></p><p>Read more: <a href="https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_lock" target="_blank" rel="noopener">proxy_cache_lock</a></p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>As you can see, with a few lines of code, we can easily add caching to your services. While the cache only works at the public endpoint level, it should act as a good starting point and later Redis etc can be added for caching intermediate calculations within your service. Since it’s a separate proxy, it can also be added between your backend and legacy or external services.</p><p>Thanks for reading! Feel free to follow me in <a href="https://twitter.com/sheshbabu" target="_blank" rel="noopener">Twitter</a> for more posts like this :)</p><h3 id="References"><a href="#References" class="headerlink" title="References"></a>References</h3><ul><li><a href="https://nginx.org/en/docs/http/ngx_http_proxy_module.html" target="_blank" rel="noopener">https://nginx.org/en/docs/http/ngx_http_proxy_module.html</a></li><li><a href="https://docs.nginx.com/nginx/admin-guide/content-cache/content-caching/" target="_blank" rel="noopener">https://docs.nginx.com/nginx/admin-guide/content-cache/content-caching/</a></li><li><a href="https://www.nginx.com/blog/nginx-caching-guide" target="_blank" rel="noopener">https://www.nginx.com/blog/nginx-caching-guide</a></li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;Most of us are familiar with Nginx - it’s a very popular Web Server and Reverse Proxy. But do you know that you can also use it as a Cach
      
    
    </summary>
    
    
      <category term="Nginx" scheme="https://www.sheshbabu.com/tags/Nginx/"/>
    
      <category term="Cache" scheme="https://www.sheshbabu.com/tags/Cache/"/>
    
      <category term="Reliability" scheme="https://www.sheshbabu.com/tags/Reliability/"/>
    
  </entry>
  
  <entry>
    <title>Single Page Applications using Rust</title>
    <link href="https://www.sheshbabu.com/posts/rust-wasm-yew-single-page-application/"/>
    <id>https://www.sheshbabu.com/posts/rust-wasm-yew-single-page-application/</id>
    <published>2020-08-11T03:30:00.000Z</published>
    <updated>2024-05-19T06:17:28.272Z</updated>
    
    <content type="html"><![CDATA[<p>WebAssembly (wasm) allows code written in languages other than JavaScript to run on browsers. If you haven’t been paying attention, all the major browsers support wasm and <a href="https://caniuse.com/#feat=wasm" target="_blank" rel="noopener">globally more than 90% of users</a> have browsers that can run wasm.</p><p>Since Rust compiles to wasm, is it possible to build SPAs (Single Page Applications) purely in Rust and without writing a single line of JavaScript? The short answer is YES! Read on to learn more or visit the <a href="https://rustmart-yew.netlify.app" target="_blank" rel="noopener">demo site</a> if you can’t contain your excitement!</p><p>We’ll be building a simple ecommerce site called “RustMart” that will have 2 pages:</p><ul><li>HomePage - list all the products that the customer can add to cart</li><li>ProductDetailPage - show the product details when a product card is clicked</li></ul><p><img src="/images/2020-rust-wasm-yew-single-page-application/rust-wasm-yew-single-page-application-1.png" alt=""></p><p>I’m using this example as it tests the minimal set of capabilities required to build modern SPAs:</p><ul><li>Navigate between multiple pages without page reload</li><li>Make network requests without page reload</li><li>Ability to reuse UI components across multiple pages</li><li>Update components in different layers of the UI hierarchy</li></ul><h2 id="Setup"><a href="#Setup" class="headerlink" title="Setup"></a>Setup</h2><p>Follow this <a href="https://www.rust-lang.org/tools/install" target="_blank" rel="noopener">link</a> to install Rust if you haven’t done so already.</p><p>Install these Rust tools:</p><pre class=" language-shell"><code class="language-shell">$ cargo install wasm-pack          # Compile Rust to Wasm and generate JS interop code$ cargo install cargo-make         # Task runner$ cargo install simple-http-server # Simple server to serve assets</code></pre><p>Create a new project:</p><pre class=" language-shell"><code class="language-shell">$ cargo new --lib rustmart && cd rustmart</code></pre><p>We’ll be using the <a href="https://yew.rs" target="_blank" rel="noopener"><code>Yew</code></a> library to build UI components. Let’s add this and wasm dependencies to <code>Cargo.toml</code>:</p><pre class=" language-toml"><code class="language-toml">[lib]crate-type = ["cdylib", "rlib"][dependencies]yew = "0.17"wasm-bindgen = "0.2"</code></pre><p>Create a new file named <code>Makefile.toml</code> and add this:</p><pre class=" language-toml"><code class="language-toml">[tasks.build]command = "wasm-pack"args = ["build", "--dev", "--target", "web", "--out-name", "wasm", "--out-dir", "./static"]watch = { ignore_pattern = "static/*" }[tasks.serve]command = "simple-http-server"args = ["-i", "./static/", "-p", "3000", "--nocache", "--try-file", "./static/index.html"]</code></pre><p>Start the build task:</p><pre class=" language-shell"><code class="language-shell">$ cargo make build</code></pre><p>If you’re new to Rust, I’ve written some <a href="http://www.sheshbabu.com/tags/Rust-Beginners/">guides for beginners</a> which will help you follow this post better.</p><h2 id="Hello-World"><a href="#Hello-World" class="headerlink" title="Hello World"></a>Hello World</h2><p>Let’s start with a simple “hello world” example:</p><p>Create <code>static/index.html</code> and add this:</p><pre class=" language-html"><code class="language-html"><span class="token doctype">&lt;!DOCTYPE html></span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>html</span> <span class="token attr-name">lang</span><span class="token attr-value"><span class="token punctuation">=</span><span class="token punctuation">"</span>en<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>  <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>head</span><span class="token punctuation">></span></span>    <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>meta</span> <span class="token attr-name">charset</span><span class="token attr-value"><span class="token punctuation">=</span><span class="token punctuation">"</span>utf-8<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span>    <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>title</span><span class="token punctuation">></span></span>RustMart<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>title</span><span class="token punctuation">></span></span>    <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>script</span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation">=</span><span class="token punctuation">"</span>module<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token script language-javascript">      <span class="token keyword">import</span> init <span class="token keyword">from</span> <span class="token string">"/wasm.js"</span><span class="token punctuation">;</span>      <span class="token function">init</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    </span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>script</span><span class="token punctuation">></span></span>    <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>link</span> <span class="token attr-name">rel</span><span class="token attr-value"><span class="token punctuation">=</span><span class="token punctuation">"</span>shortcut icon<span class="token punctuation">"</span></span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation">=</span><span class="token punctuation">"</span>#<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span>  <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>head</span><span class="token punctuation">></span></span>  <span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>body</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>body</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>html</span><span class="token punctuation">></span></span></code></pre><p>Add this to <code>src/lib.rs</code>:</p><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// src/lib.rs</span><span class="token keyword">use</span> wasm_bindgen<span class="token punctuation">:</span><span class="token punctuation">:</span>prelude<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token operator">*</span><span class="token punctuation">;</span><span class="token keyword">use</span> yew<span class="token punctuation">:</span><span class="token punctuation">:</span>prelude<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token operator">*</span><span class="token punctuation">;</span><span class="token keyword">struct</span> Hello <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token keyword">impl</span> Component <span class="token keyword">for</span> Hello <span class="token punctuation">{</span>    <span class="token keyword">type</span> Message <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token keyword">type</span> Properties <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token keyword">fn</span> <span class="token function">create</span><span class="token punctuation">(</span>_<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Properties<span class="token punctuation">,</span> _<span class="token punctuation">:</span> ComponentLink<span class="token operator">&lt;</span>Self<span class="token operator">></span><span class="token punctuation">)</span> <span class="token punctuation">-></span> Self <span class="token punctuation">{</span>        Self <span class="token punctuation">{</span><span class="token punctuation">}</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">update</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">mut</span> <span class="token keyword">self</span><span class="token punctuation">,</span> _<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Message<span class="token punctuation">)</span> <span class="token punctuation">-></span> ShouldRender <span class="token punctuation">{</span>        <span class="token keyword">true</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">change</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">mut</span> <span class="token keyword">self</span><span class="token punctuation">,</span> _<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Properties<span class="token punctuation">)</span> <span class="token punctuation">-></span> ShouldRender <span class="token punctuation">{</span>        <span class="token keyword">true</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">view</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">self</span><span class="token punctuation">)</span> <span class="token punctuation">-></span> Html <span class="token punctuation">{</span>        <span class="token macro-rules function">html!</span> <span class="token punctuation">{</span> <span class="token operator">&lt;</span>span<span class="token operator">></span><span class="token punctuation">{</span><span class="token string">"Hello World!"</span><span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>span<span class="token operator">></span> <span class="token punctuation">}</span>    <span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token attribute attr-name">#[wasm_bindgen(start)]</span><span class="token keyword">pub</span> <span class="token keyword">fn</span> <span class="token function">run_app</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    App<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token operator">&lt;</span>Hello<span class="token operator">></span><span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">new</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">mount_to_body</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>Lot of things going on but you can see that we’re creating a new component named “Hello” that renders <code>&lt;span&gt;Hello World!&lt;/span&gt;</code> into the DOM. We’ll learn more about Yew components later.</p><p>Start the serve task in a new terminal and load <code>http://localhost:3000</code> in your browser</p><pre class=" language-shell"><code class="language-shell">$ cargo make serve</code></pre><p><img src="/images/2020-rust-wasm-yew-single-page-application/image-1.png" alt=""></p><p>It works!! It’s only “hello world” but this is fully written in Rust.</p><p>Let’s learn about components and other SPA concepts before proceeding further.</p><h2 id="Thinking-in-Components"><a href="#Thinking-in-Components" class="headerlink" title="Thinking in Components"></a>Thinking in Components</h2><p>Building UIs by composing components and passing data in a unidirectional way is a paradigm shift in the frontend world. It’s a huge improvement in the way we reason about UI and it’s very hard to go back to imperative DOM manipulation once you get used to this.</p><p>A <code>Component</code> in libraries like React, Vue, Yew, Flutter etc have these features:</p><ul><li>Ability to be composed into bigger components</li><li><code>Props</code> - Pass data and callbacks from that component to its child components.</li><li><code>State</code> - Manipulate state local to that component.</li><li><code>AppState</code> - Manipulate global state.</li><li>Listen to lifecycle events like “Instantiated”, “Mounted in DOM” etc</li><li>Perform side effects like fetching remote data, manipulating localstorage etc</li></ul><p>A component gets updated (re-rendered) when one of the following happens:</p><ul><li>Parent component is re-rendered</li><li><code>Props</code> changes</li><li><code>State</code> changes</li><li><code>AppState</code> changes</li></ul><p>So, instead of imperatively updating the UI when user interaction, network requests etc happen, we update the data (Props, State, AppState) and the UI is updated based on this data. This what someone means when they say “UI is a function of state”.</p><p>The exact details differ across different libraries but this should give you a general idea. If you’re new to this, this way of thinking might take sometime to “click” and get used to.</p><h2 id="HomePage"><a href="#HomePage" class="headerlink" title="HomePage"></a>HomePage</h2><p>Let’s build the HomePage first. We’ll be building the HomePage as a monolithic component and later decompose it into smaller reusable components.</p><p>Let’s create the following files:</p><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// src/pages/home.rs</span><span class="token keyword">use</span> yew<span class="token punctuation">:</span><span class="token punctuation">:</span>prelude<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token operator">*</span><span class="token punctuation">;</span><span class="token keyword">pub</span> <span class="token keyword">struct</span> Home <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token keyword">impl</span> Component <span class="token keyword">for</span> Home <span class="token punctuation">{</span>    <span class="token keyword">type</span> Message <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token keyword">type</span> Properties <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token keyword">fn</span> <span class="token function">create</span><span class="token punctuation">(</span>_<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Properties<span class="token punctuation">,</span> _<span class="token punctuation">:</span> ComponentLink<span class="token operator">&lt;</span>Self<span class="token operator">></span><span class="token punctuation">)</span> <span class="token punctuation">-></span> Self <span class="token punctuation">{</span>        Self <span class="token punctuation">{</span><span class="token punctuation">}</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">update</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">mut</span> <span class="token keyword">self</span><span class="token punctuation">,</span> _<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Message<span class="token punctuation">)</span> <span class="token punctuation">-></span> ShouldRender <span class="token punctuation">{</span>        <span class="token keyword">true</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">change</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">mut</span> <span class="token keyword">self</span><span class="token punctuation">,</span> _<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Properties<span class="token punctuation">)</span> <span class="token punctuation">-></span> ShouldRender <span class="token punctuation">{</span>        <span class="token keyword">true</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">view</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">self</span><span class="token punctuation">)</span> <span class="token punctuation">-></span> Html <span class="token punctuation">{</span>        <span class="token macro-rules function">html!</span> <span class="token punctuation">{</span> <span class="token operator">&lt;</span>span<span class="token operator">></span><span class="token punctuation">{</span><span class="token string">"Home Sweet Home!"</span><span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>span<span class="token operator">></span> <span class="token punctuation">}</span>    <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// src/pages/mod.rs</span><span class="token keyword">mod</span> home<span class="token punctuation">;</span><span class="token keyword">pub</span> <span class="token keyword">use</span> home<span class="token punctuation">:</span><span class="token punctuation">:</span>Home<span class="token punctuation">;</span></code></pre><p>Let’s update the <code>src/lib.rs</code> to import the HomePage component:</p><pre class=" language-diff"><code class="language-diff">  // src/lib.rs<span class="token inserted">+ mod pages;</span><span class="token inserted">+ use pages::Home;</span>  use wasm_bindgen::prelude::*;  use yew::prelude::*;<span class="token deleted">- struct Hello {}</span><span class="token deleted">- impl Component for Hello {</span><span class="token deleted">-     type Message = ();</span><span class="token deleted">-     type Properties = ();</span><span class="token deleted">-     fn create(_: Self::Properties, _: ComponentLink&lt;Self>) -> Self {</span><span class="token deleted">-         Self {}</span><span class="token deleted">-     }</span><span class="token deleted">-     fn update(&amp;mut self, _: Self::Message) -> ShouldRender {</span><span class="token deleted">-         true</span><span class="token deleted">-     }</span><span class="token deleted">-     fn change(&amp;mut self, _: Self::Properties) -> ShouldRender {</span><span class="token deleted">-         true</span><span class="token deleted">-     }</span><span class="token deleted">-     fn view(&amp;self) -> Html {</span><span class="token deleted">-         html! { &lt;span>{"Hello World!"}&lt;/span> }</span><span class="token deleted">-     }</span><span class="token deleted">- }</span>  #[wasm_bindgen(start)]  pub fn run_app() {<span class="token deleted">-   App::&lt;Hello>::new().mount_to_body();</span><span class="token inserted">+   App::&lt;Home>::new().mount_to_body();</span>  }</code></pre><p>Now, you should see “Home Sweet Home!” instead of “Hello World!” rendered in your browser.</p><p>Let’s start designing the <code>State</code> of this component:</p><ul><li>We need to store a list of products retrieved from server</li><li>Store the products the user has added to cart</li></ul><p>We create a simple struct to hold the <code>Product</code> details:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">struct</span> Product <span class="token punctuation">{</span>    name<span class="token punctuation">:</span> String<span class="token punctuation">,</span>    description<span class="token punctuation">:</span> String<span class="token punctuation">,</span>    image<span class="token punctuation">:</span> String<span class="token punctuation">,</span>    price<span class="token punctuation">:</span> f64<span class="token punctuation">,</span><span class="token punctuation">}</span></code></pre><p>We then create a new struct <code>State</code> with field called <code>products</code> to hold the products from server:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">struct</span> State <span class="token punctuation">{</span>    products<span class="token punctuation">:</span> Vec<span class="token operator">&lt;</span>Product<span class="token operator">></span><span class="token punctuation">,</span><span class="token punctuation">}</span></code></pre><p>Here’s the full list of changes in the HomePage component:</p><pre class=" language-diff"><code class="language-diff">  use yew::prelude::*;<span class="token inserted">+ struct Product {</span><span class="token inserted">+     id: i32,</span><span class="token inserted">+     name: String,</span><span class="token inserted">+     description: String,</span><span class="token inserted">+     image: String,</span><span class="token inserted">+     price: f64,</span><span class="token inserted">+ }</span><span class="token inserted">+ struct State {</span><span class="token inserted">+     products: Vec&lt;Product>,</span><span class="token inserted">+ }</span><span class="token deleted">- pub struct Home {}</span><span class="token inserted">+ pub struct Home {</span><span class="token inserted">+     state: State,</span><span class="token inserted">+ }</span>  impl Component for Home {      type Message = ();      type Properties = ();      fn create(_: Self::Properties, _: ComponentLink&lt;Self>) -> Self {<span class="token inserted">+       let products: Vec&lt;Product> = vec![</span><span class="token inserted">+           Product {</span><span class="token inserted">+               id: 1,</span><span class="token inserted">+               name: "Apple".to_string(),</span><span class="token inserted">+               description: "An apple a day keeps the doctor away".to_string(),</span><span class="token inserted">+               image: "/products/apple.png".to_string(),</span><span class="token inserted">+               price: 3.65,</span><span class="token inserted">+           },</span><span class="token inserted">+           Product {</span><span class="token inserted">+               id: 2,</span><span class="token inserted">+               name: "Banana".to_string(),</span><span class="token inserted">+               description: "An old banana leaf was once young and green".to_string(),</span><span class="token inserted">+               image: "/products/banana.png".to_string(),</span><span class="token inserted">+               price: 7.99,</span><span class="token inserted">+           },</span><span class="token inserted">+       ];</span><span class="token deleted">-       Self {}</span><span class="token inserted">+       Self {</span><span class="token inserted">+           state: State {</span><span class="token inserted">+               products,</span><span class="token inserted">+           },</span><span class="token inserted">+       }</span>      }      fn update(&amp;mut self, _: Self::Message) -> ShouldRender {          true      }      fn change(&amp;mut self, _: Self::Properties) -> ShouldRender {          true      }      fn view(&amp;self) -> Html {<span class="token inserted">+        let products: Vec&lt;Html> = self</span><span class="token inserted">+            .state</span><span class="token inserted">+            .products</span><span class="token inserted">+            .iter()</span><span class="token inserted">+            .map(|product: &amp;Product| {</span><span class="token inserted">+                html! {</span><span class="token inserted">+                  &lt;div></span><span class="token inserted">+                    &lt;img src={&amp;product.image}/></span><span class="token inserted">+                    &lt;div>{&amp;product.name}&lt;/div></span><span class="token inserted">+                    &lt;div>{"$"}{&amp;product.price}&lt;/div></span><span class="token inserted">+                  &lt;/div></span><span class="token inserted">+                }</span><span class="token inserted">+            })</span><span class="token inserted">+            .collect();</span><span class="token inserted">+</span><span class="token inserted">+        html! { &lt;span>{products}&lt;/span> }</span><span class="token deleted">-        html! { &lt;span>{"Home!"}&lt;/span> }</span>      }  }</code></pre><p>The <code>create</code> lifecycle method is invoked when the component is created and this is where we set the initial state. For the time being, we’ve created a mock list of products and assigned it to the <code>products</code> inside the state as initial value. Later, we’ll fetch this list using network request.</p><p>The <code>view</code> lifecycle method is invoked when the component is rendered. Here we’ve iterated over <code>products</code> inside state to generate product cards. If you’re familiar with React, this is same as the <code>render</code> method and the <code>html!</code> macro is similar to <code>JSX</code>.</p><p>Save some random images as <code>static/products/apple.png</code> and <code>static/products/banana.png</code> and you’ll get this UI:</p><p><img src="/images/2020-rust-wasm-yew-single-page-application/image-2.png" alt=""></p><p>Let’s implement the “add to cart” functionality:</p><ul><li>We keep track of all products added to cart in a new state field called <code>cart_products</code></li><li>We render a “add to cart” button for each product</li><li>Add logic to update the <code>cart_products</code> state when “add to cart” button is clicked</li></ul><pre class=" language-diff"><code class="language-diff">  use yew::prelude::*;<span class="token inserted">+ #[derive(Clone)]</span>  struct Product {      id: i32,      name: String,      description: String,      image: String,      price: f64,  }<span class="token inserted">+ struct CartProduct {</span><span class="token inserted">+     product: Product,</span><span class="token inserted">+     quantity: i32,</span><span class="token inserted">+ }</span>  struct State {      products: Vec&lt;Product>,<span class="token inserted">+     cart_products: Vec&lt;CartProduct>,</span>  }  pub struct Home {      state: State,<span class="token inserted">+     link: ComponentLink&lt;Self>,</span>  }<span class="token inserted">+ pub enum Msg {</span><span class="token inserted">+     AddToCart(i32),</span><span class="token inserted">+ }</span>  impl Component for Home {<span class="token deleted">-   type Message = ();</span><span class="token inserted">+   type Message = Msg;</span>    type Properties = ();<span class="token deleted">-   fn create(_: Self::Properties, _: ComponentLink&lt;Self>) -> Self {</span><span class="token inserted">+   fn create(_: Self::Properties, link: ComponentLink&lt;Self>) -> Self {</span>        let products: Vec&lt;Product> = vec![            Product {                id: 1,                name: "Apple".to_string(),                description: "An apple a day keeps the doctor away".to_string(),                image: "/products/apple.png".to_string(),                price: 3.65,            },            Product {                id: 2,                name: "Banana".to_string(),                description: "An old banana leaf was once young and green".to_string(),                image: "/products/banana.png".to_string(),                price: 7.99,            },        ];<span class="token inserted">+       let cart_products = vec![];</span>        Self {            state: State {                products,<span class="token inserted">+               cart_products,</span>            },<span class="token inserted">+           link,</span>        }    }<span class="token deleted">-   fn update(&amp;mut self, _: Self::Message) -> ShouldRender {</span><span class="token inserted">+   fn update(&amp;mut self, message: Self::Message) -> ShouldRender {</span><span class="token inserted">+       match message {</span><span class="token inserted">+           Msg::AddToCart(product_id) => {</span><span class="token inserted">+               let product = self</span><span class="token inserted">+                   .state</span><span class="token inserted">+                   .products</span><span class="token inserted">+                   .iter()</span><span class="token inserted">+                   .find(|p: &amp;&amp;Product| p.id == product_id)</span><span class="token inserted">+                   .unwrap();</span><span class="token inserted">+               let cart_product = self</span><span class="token inserted">+                   .state</span><span class="token inserted">+                   .cart_products</span><span class="token inserted">+                   .iter_mut()</span><span class="token inserted">+                   .find(|cp: &amp;&amp;mut CartProduct| cp.product.id == product_id);</span><span class="token inserted">+</span><span class="token inserted">+               if let Some(cp) = cart_product {</span><span class="token inserted">+                   cp.quantity += 1;</span><span class="token inserted">+               } else {</span><span class="token inserted">+                   self.state.cart_products.push(CartProduct {</span><span class="token inserted">+                       product: product.clone(),</span><span class="token inserted">+                       quantity: 1,</span><span class="token inserted">+                   })</span><span class="token inserted">+               }</span><span class="token inserted">+               true</span><span class="token inserted">+           }</span><span class="token inserted">+       }</span><span class="token deleted">-       true</span>    }    fn change(&amp;mut self, _: Self::Properties) -> ShouldRender {        true    }    fn view(&amp;self) -> Html {        let products: Vec&lt;Html> = self            .state            .products            .iter()            .map(|product: &amp;Product| {<span class="token inserted">+              let product_id = product.id;</span>                html! {                  &lt;div>                    &lt;img src={&amp;product.image}/>                    &lt;div>{&amp;product.name}&lt;/div>                    &lt;div>{"$"}{&amp;product.price}&lt;/div><span class="token inserted">+                   &lt;button onclick=self.link.callback(move |_| Msg::AddToCart(product_id))>{"Add To Cart"}&lt;/button></span>                  &lt;/div>                }            })            .collect();<span class="token inserted">+       let cart_value = self</span><span class="token inserted">+           .state</span><span class="token inserted">+           .cart_products</span><span class="token inserted">+           .iter()</span><span class="token inserted">+           .fold(0.0, |acc, cp| acc + (cp.quantity as f64 * cp.product.price));</span><span class="token deleted">-       html! { &lt;span>{products}&lt;/span> }</span><span class="token inserted">+       html! {</span><span class="token inserted">+         &lt;div></span><span class="token inserted">+           &lt;span>{format!("Cart Value: {:.2}", cart_value)}&lt;/span></span><span class="token inserted">+           &lt;span>{products}&lt;/span></span><span class="token inserted">+         &lt;/div></span><span class="token inserted">+       }</span>      }  }</code></pre><ul><li><code>clone</code> - We’ve derived the <a href="https://doc.rust-lang.org/std/clone/trait.Clone.html" target="_blank" rel="noopener"><code>Clone</code></a> trait in <code>Product</code> struct so we can save the cloned <code>Product</code> into <code>CartProduct</code> whenever the user adds them to cart.</li><li><code>update</code> - This method is the place where the logic to update the component <code>State</code> or perform side-effects (like network requests) exist. It is invoked using a <code>Message</code> enum that contains all the actions the component supports. When we return <code>true</code> from this method, the component is re-rendered. In the above code, when the user clicks the “Add To Cart” button, we send a <code>Msg::AddToCart</code> message to <code>update</code>. Inside <code>update</code>, this either adds the product to <code>cart_product</code> if it doesn’t exist or it increments the quantity.</li><li><code>link</code> - This allows us to register callbacks that can trigger our <code>update</code> lifecycle method.</li></ul><p>If you’ve used <a href="https://redux.js.org" target="_blank" rel="noopener">Redux</a> before, <code>update</code> is similar to <a href="https://redux.js.org/basics/reducers" target="_blank" rel="noopener"><code>Reducer</code></a> (for state updates) and <a href="https://redux.js.org/basics/actions#action-creators" target="_blank" rel="noopener"><code>Action Creator</code></a> (for side-effects), <code>Message</code> is similar to <a href="https://redux.js.org/basics/actions" target="_blank" rel="noopener"><code>Action</code></a> and <code>link</code> is similar to <a href="https://redux.js.org/basics/store#dispatching-actions" target="_blank" rel="noopener"><code>Dispatch</code></a>.</p><p><img src="/images/2020-rust-wasm-yew-single-page-application/rust-wasm-yew-single-page-application-2.png" alt=""></p><p>Here’s how the UI looks like, try clicking the “Add To Cart” button and see the changes in “Cart Value”:</p><p><img src="/images/2020-rust-wasm-yew-single-page-application/image-3.png" alt=""></p><h2 id="Fetching-Data"><a href="#Fetching-Data" class="headerlink" title="Fetching Data"></a>Fetching Data</h2><p>We’ll move the product data from the <code>create</code> function to <code>static/products/products.json</code> and query it using the <a href="https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API" target="_blank" rel="noopener"><code>fetch</code></a> api.</p><pre class=" language-json"><code class="language-json"><span class="token punctuation">[</span>  <span class="token punctuation">{</span>    <span class="token property">"id"</span><span class="token operator">:</span> <span class="token number">1</span><span class="token punctuation">,</span>    <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"Apple"</span><span class="token punctuation">,</span>    <span class="token property">"description"</span><span class="token operator">:</span> <span class="token string">"An apple a day keeps the doctor away"</span><span class="token punctuation">,</span>    <span class="token property">"image"</span><span class="token operator">:</span> <span class="token string">"/products/apple.png"</span><span class="token punctuation">,</span>    <span class="token property">"price"</span><span class="token operator">:</span> <span class="token number">3.65</span>  <span class="token punctuation">}</span><span class="token punctuation">,</span>  <span class="token punctuation">{</span>    <span class="token property">"id"</span><span class="token operator">:</span> <span class="token number">2</span><span class="token punctuation">,</span>    <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"Banana"</span><span class="token punctuation">,</span>    <span class="token property">"description"</span><span class="token operator">:</span> <span class="token string">"An old banana leaf was once young and green"</span><span class="token punctuation">,</span>    <span class="token property">"image"</span><span class="token operator">:</span> <span class="token string">"/products/banana.png"</span><span class="token punctuation">,</span>    <span class="token property">"price"</span><span class="token operator">:</span> <span class="token number">7.99</span>  <span class="token punctuation">}</span><span class="token punctuation">]</span></code></pre><p>Yew exposes common browser apis like fetch, localstorage etc through something called <a href="https://docs.rs/yew/0.17.2/yew/services/index.html" target="_blank" rel="noopener">“services”</a>. We can use the <code>FetchService</code> to make network requests. It requires <code>anyhow</code> and <code>serde</code> crates, let’s install them:</p><pre class=" language-diff"><code class="language-diff">  [package]  name = "rustmart"  version = "0.1.0"  authors = ["sheshbabu &lt;sheshbabu@gmail.com>"]  edition = "2018"  [lib]  crate-type = ["cdylib", "rlib"]  [dependencies]  yew = "0.17"  wasm-bindgen = "0.2"<span class="token inserted">+ anyhow = "1.0.32"</span><span class="token inserted">+ serde = { version = "1.0", features = ["derive"] }</span></code></pre><p>Let’s extract the <code>Product</code> and <code>CartProduct</code> to <code>src/types.rs</code> so we can share it across multiple files:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">use</span> serde<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token punctuation">{</span>Deserialize<span class="token punctuation">,</span> Serialize<span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token attribute attr-name">#[derive(Deserialize, Serialize, Clone, Debug)]</span><span class="token keyword">pub</span> <span class="token keyword">struct</span> Product <span class="token punctuation">{</span>    <span class="token keyword">pub</span> id<span class="token punctuation">:</span> i32<span class="token punctuation">,</span>    <span class="token keyword">pub</span> name<span class="token punctuation">:</span> String<span class="token punctuation">,</span>    <span class="token keyword">pub</span> description<span class="token punctuation">:</span> String<span class="token punctuation">,</span>    <span class="token keyword">pub</span> image<span class="token punctuation">:</span> String<span class="token punctuation">,</span>    <span class="token keyword">pub</span> price<span class="token punctuation">:</span> f64<span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token attribute attr-name">#[derive(Clone, Debug)]</span><span class="token keyword">pub</span> <span class="token keyword">struct</span> CartProduct <span class="token punctuation">{</span>    <span class="token keyword">pub</span> product<span class="token punctuation">:</span> Product<span class="token punctuation">,</span>    <span class="token keyword">pub</span> quantity<span class="token punctuation">:</span> i32<span class="token punctuation">,</span><span class="token punctuation">}</span></code></pre><p>We’ve made both structs and their fields public, and have derived the <code>Deserialize</code> and <code>Serialize</code> traits.</p><p>We’ll use the <a href="http://www.sheshbabu.com/posts/organizing-http-requests-using-api-module-pattern/">API module pattern</a> and create a separate module called <code>src/api.rs</code> to hold our fetch logic:</p><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// src/api.rs</span><span class="token keyword">use</span> <span class="token keyword">crate</span><span class="token punctuation">:</span><span class="token punctuation">:</span>types<span class="token punctuation">:</span><span class="token punctuation">:</span>Product<span class="token punctuation">;</span><span class="token keyword">use</span> anyhow<span class="token punctuation">:</span><span class="token punctuation">:</span>Error<span class="token punctuation">;</span><span class="token keyword">use</span> yew<span class="token punctuation">:</span><span class="token punctuation">:</span>callback<span class="token punctuation">:</span><span class="token punctuation">:</span>Callback<span class="token punctuation">;</span><span class="token keyword">use</span> yew<span class="token punctuation">:</span><span class="token punctuation">:</span>format<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token punctuation">{</span>Json<span class="token punctuation">,</span> Nothing<span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token keyword">use</span> yew<span class="token punctuation">:</span><span class="token punctuation">:</span>services<span class="token punctuation">:</span><span class="token punctuation">:</span>fetch<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token punctuation">{</span>FetchService<span class="token punctuation">,</span> FetchTask<span class="token punctuation">,</span> Request<span class="token punctuation">,</span> Response<span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token keyword">pub</span> <span class="token keyword">type</span> FetchResponse<span class="token operator">&lt;</span>T<span class="token operator">></span> <span class="token operator">=</span> Response<span class="token operator">&lt;</span>Json<span class="token operator">&lt;</span>Result<span class="token operator">&lt;</span>T<span class="token punctuation">,</span> Error<span class="token operator">>></span><span class="token operator">></span><span class="token punctuation">;</span><span class="token keyword">type</span> FetchCallback<span class="token operator">&lt;</span>T<span class="token operator">></span> <span class="token operator">=</span> Callback<span class="token operator">&lt;</span>FetchResponse<span class="token operator">&lt;</span>T<span class="token operator">>></span><span class="token punctuation">;</span><span class="token keyword">pub</span> <span class="token keyword">fn</span> <span class="token function">get_products</span><span class="token punctuation">(</span>callback<span class="token punctuation">:</span> FetchCallback<span class="token operator">&lt;</span>Vec<span class="token operator">&lt;</span>Product<span class="token operator">>></span><span class="token punctuation">)</span> <span class="token punctuation">-></span> FetchTask <span class="token punctuation">{</span>    <span class="token keyword">let</span> req <span class="token operator">=</span> Request<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">get</span><span class="token punctuation">(</span><span class="token string">"/products/products.json"</span><span class="token punctuation">)</span>        <span class="token punctuation">.</span><span class="token function">body</span><span class="token punctuation">(</span>Nothing<span class="token punctuation">)</span>        <span class="token punctuation">.</span><span class="token function">unwrap</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    FetchService<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">fetch</span><span class="token punctuation">(</span>req<span class="token punctuation">,</span> callback<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">unwrap</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">}</span></code></pre><p>The <code>FetchService</code> api is a bit awkward - it takes in a request object and callback as arguments and returns something called a “FetchTask”. One surprising gotcha here is that the network request gets aborted if this “FetchTask” is dropped. So we return this and store it in our component.</p><p>Let’s update <code>lib.rs</code> to add these new modules into the <a href="http://www.sheshbabu.com/posts/rust-module-system/">module tree</a>:</p><pre class=" language-diff"><code class="language-diff">  // src/lib.rs<span class="token inserted">+ mod api;</span><span class="token inserted">+ mod types;</span>  mod pages;  use pages::Home;  use wasm_bindgen::prelude::*;  use yew::prelude::*;  #[wasm_bindgen(start)]  pub fn run_app() {      App::&lt;Home>::new().mount_to_body();  }</code></pre><p>Finally, let’s update our HomePage component:</p><pre class=" language-diff"><code class="language-diff"><span class="token inserted">+ use crate::api;</span><span class="token inserted">+ use crate::types::{CartProduct, Product};</span><span class="token inserted">+ use anyhow::Error;</span><span class="token inserted">+ use yew::format::Json;</span><span class="token inserted">+ use yew::services::fetch::FetchTask;</span>  use yew::prelude::*;<span class="token deleted">- #[derive(Clone)]</span><span class="token deleted">- struct Product {</span><span class="token deleted">-     id: i32,</span><span class="token deleted">-     name: String,</span><span class="token deleted">-     description: String,</span><span class="token deleted">-     image: String,</span><span class="token deleted">-     price: f64,</span><span class="token deleted">- }</span><span class="token deleted">- struct CartProduct {</span><span class="token deleted">-     product: Product,</span><span class="token deleted">-     quantity: i32,</span><span class="token deleted">- }</span>  struct State {      products: Vec&lt;Product>,      cart_products: Vec&lt;CartProduct>,<span class="token inserted">+     get_products_error: Option&lt;Error>,</span><span class="token inserted">+     get_products_loaded: bool,</span>  }  pub struct Home {      state: State,      link: ComponentLink&lt;Self>,<span class="token inserted">+     task: Option&lt;FetchTask>,</span>  }  pub enum Msg {      AddToCart(i32),<span class="token inserted">+     GetProducts,</span><span class="token inserted">+     GetProductsSuccess(Vec&lt;Product>),</span><span class="token inserted">+     GetProductsError(Error),</span>  }  impl Component for Home {      type Message = Msg;      type Properties = ();      fn create(_: Self::Properties, link: ComponentLink&lt;Self>) -> Self {<span class="token deleted">-         let products: Vec&lt;Product> = vec![</span><span class="token deleted">-             Product {</span><span class="token deleted">-                 id: 1,</span><span class="token deleted">-                 name: "Apple".to_string(),</span><span class="token deleted">-                 description: "An apple a day keeps the doctor away".to_string(),</span><span class="token deleted">-                 image: "/products/apple.png".to_string(),</span><span class="token deleted">-                 price: 3.65,</span><span class="token deleted">-             },</span><span class="token deleted">-             Product {</span><span class="token deleted">-                 id: 2,</span><span class="token deleted">-                 name: "Banana".to_string(),</span><span class="token deleted">-                 description: "An old banana leaf was once young and green".to_string(),</span><span class="token deleted">-                 image: "/products/banana.png".to_string(),</span><span class="token deleted">-                 price: 7.99,</span><span class="token deleted">-             },</span><span class="token deleted">-         ];</span><span class="token inserted">+         let products = vec![];</span>          let cart_products = vec![];<span class="token inserted">+         link.send_message(Msg::GetProducts);</span>          Self {              state: State {                  products,                  cart_products,<span class="token inserted">+                 get_products_error: None,</span><span class="token inserted">+                 get_products_loaded: false,</span>              },              link,<span class="token inserted">+             task: None,</span>          }      }      fn update(&amp;mut self, message: Self::Message) -> ShouldRender {          match message {<span class="token inserted">+             Msg::GetProducts => {</span><span class="token inserted">+                 self.state.get_products_loaded = false;</span><span class="token inserted">+                 let handler =</span><span class="token inserted">+                     self.link</span><span class="token inserted">+                         .callback(move |response: api::FetchResponse&lt;Vec&lt;Product>>| {</span><span class="token inserted">+                             let (_, Json(data)) = response.into_parts();</span><span class="token inserted">+                             match data {</span><span class="token inserted">+                                 Ok(products) => Msg::GetProductsSuccess(products),</span><span class="token inserted">+                                 Err(err) => Msg::GetProductsError(err),</span><span class="token inserted">+                             }</span><span class="token inserted">+                         });</span><span class="token inserted">+                 self.task = Some(api::get_products(handler));</span><span class="token inserted">+                 true</span><span class="token inserted">+             }</span><span class="token inserted">+             Msg::GetProductsSuccess(products) => {</span><span class="token inserted">+                 self.state.products = products;</span><span class="token inserted">+                 self.state.get_products_loaded = true;</span><span class="token inserted">+                 true</span><span class="token inserted">+             }</span><span class="token inserted">+             Msg::GetProductsError(error) => {</span><span class="token inserted">+                 self.state.get_products_error = Some(error);</span><span class="token inserted">+                 self.state.get_products_loaded = true;</span><span class="token inserted">+                 true</span><span class="token inserted">+             }</span>              Msg::AddToCart(product_id) => {                  let product = self                      .state                      .products                      .iter()                      .find(|p: &amp;&amp;Product| p.id == product_id)                      .unwrap();                  let cart_product = self                      .state                      .cart_products                      .iter_mut()                      .find(|cp: &amp;&amp;mut CartProduct| cp.product.id == product_id);                  if let Some(cp) = cart_product {                      cp.quantity += 1;                  } else {                      self.state.cart_products.push(CartProduct {                          product: product.clone(),                          quantity: 1,                      })                  }                  true              }          }      }      fn change(&amp;mut self, _: Self::Properties) -> ShouldRender {          true      }      fn view(&amp;self) -> Html {          let products: Vec&lt;Html> = self              .state              .products              .iter()              .map(|product: &amp;Product| {                  let product_id = product.id;                  html! {                    &lt;div>                      &lt;img src={&amp;product.image}/>                      &lt;div>{&amp;product.name}&lt;/div>                      &lt;div>{"$"}{&amp;product.price}&lt;/div>                      &lt;button onclick=self.link.callback(move |_| Msg::AddToCart(product_id))>{"Add To Cart"}&lt;/button>                    &lt;/div>                  }              })              .collect();          let cart_value = self              .state              .cart_products              .iter()              .fold(0.0, |acc, cp| acc + (cp.quantity as f64 * cp.product.price));<span class="token inserted">+         if !self.state.get_products_loaded {</span><span class="token inserted">+             html! {</span><span class="token inserted">+               &lt;div>{"Loading ..."}&lt;/div></span><span class="token inserted">+             }</span><span class="token inserted">+         } else if let Some(_) = self.state.get_products_error {</span><span class="token inserted">+             html! {</span><span class="token inserted">+               &lt;div></span><span class="token inserted">+                 &lt;span>{"Error loading products! :("}&lt;/span></span><span class="token inserted">+               &lt;/div></span><span class="token inserted">+             }</span><span class="token inserted">+         } else {</span>              html! {                &lt;div>                  &lt;span>{format!("Cart Value: {:.2}", cart_value)}&lt;/span>                  &lt;span>{products}&lt;/span>                &lt;/div>              }<span class="token inserted">+         }</span>      }  }</code></pre><p>Quite a number of changes, but you should be able to understand most of them.</p><ul><li>We’ve replaced the hardcoded products list in <code>create</code> with an empty array. We’re also sending a message <code>Msg::GetProducts</code> to <code>update</code> which calls the <code>get_products</code> method in the <code>api</code> module. The returned <code>FetchTask</code> is stored in <code>task</code>.</li><li>When the network request succeeds, the <code>Msg::GetProductsSuccess</code> message is called with products list or <code>Msg::GetProductsError</code> with error.</li><li>These two messages set the <code>products</code> and <code>get_products_error</code> fields in state respectively. They also set the <code>get_products_loaded</code> state to true after the request is fulfilled.</li><li>In the <code>view</code> method, we’ve used conditional rendering to render either the loading view, error view or products view based on the component’s state.</li></ul><p><img src="/images/2020-rust-wasm-yew-single-page-application/rust-wasm-yew-single-page-application-3.png" alt=""></p><h2 id="Splitting-into-reusable-components"><a href="#Splitting-into-reusable-components" class="headerlink" title="Splitting into reusable components"></a>Splitting into reusable components</h2><p>Let’s extract the “product card” component into its own module so we can reuse it in other pages.</p><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// src/components/product_card.rs</span><span class="token keyword">use</span> <span class="token keyword">crate</span><span class="token punctuation">:</span><span class="token punctuation">:</span>types<span class="token punctuation">:</span><span class="token punctuation">:</span>Product<span class="token punctuation">;</span><span class="token keyword">use</span> yew<span class="token punctuation">:</span><span class="token punctuation">:</span>prelude<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token operator">*</span><span class="token punctuation">;</span><span class="token keyword">pub</span> <span class="token keyword">struct</span> ProductCard <span class="token punctuation">{</span>    props<span class="token punctuation">:</span> Props<span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token attribute attr-name">#[derive(Properties, Clone)]</span><span class="token keyword">pub</span> <span class="token keyword">struct</span> Props <span class="token punctuation">{</span>    <span class="token keyword">pub</span> product<span class="token punctuation">:</span> Product<span class="token punctuation">,</span>    <span class="token keyword">pub</span> on_add_to_cart<span class="token punctuation">:</span> Callback<span class="token operator">&lt;</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">></span><span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token keyword">impl</span> Component <span class="token keyword">for</span> ProductCard <span class="token punctuation">{</span>    <span class="token keyword">type</span> Message <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token keyword">type</span> Properties <span class="token operator">=</span> Props<span class="token punctuation">;</span>    <span class="token keyword">fn</span> <span class="token function">create</span><span class="token punctuation">(</span>props<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Properties<span class="token punctuation">,</span> _link<span class="token punctuation">:</span> ComponentLink<span class="token operator">&lt;</span>Self<span class="token operator">></span><span class="token punctuation">)</span> <span class="token punctuation">-></span> Self <span class="token punctuation">{</span>        Self <span class="token punctuation">{</span> props <span class="token punctuation">}</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">update</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">mut</span> <span class="token keyword">self</span><span class="token punctuation">,</span> _msg<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Message<span class="token punctuation">)</span> <span class="token punctuation">-></span> ShouldRender <span class="token punctuation">{</span>        <span class="token keyword">true</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">change</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">mut</span> <span class="token keyword">self</span><span class="token punctuation">,</span> _props<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Properties<span class="token punctuation">)</span> <span class="token punctuation">-></span> ShouldRender <span class="token punctuation">{</span>        <span class="token keyword">true</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">view</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">self</span><span class="token punctuation">)</span> <span class="token punctuation">-></span> Html <span class="token punctuation">{</span>        <span class="token keyword">let</span> onclick <span class="token operator">=</span> <span class="token keyword">self</span><span class="token punctuation">.</span>props<span class="token punctuation">.</span>on_add_to_cart<span class="token punctuation">.</span><span class="token function">reform</span><span class="token punctuation">(</span><span class="token operator">|</span>_<span class="token operator">|</span> <span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>        <span class="token macro-rules function">html!</span> <span class="token punctuation">{</span>          <span class="token operator">&lt;</span>div<span class="token operator">></span>            <span class="token operator">&lt;</span>img src<span class="token operator">=</span><span class="token punctuation">{</span><span class="token operator">&amp;</span><span class="token keyword">self</span><span class="token punctuation">.</span>props<span class="token punctuation">.</span>product<span class="token punctuation">.</span>image<span class="token punctuation">}</span><span class="token operator">/</span><span class="token operator">></span>            <span class="token operator">&lt;</span>div<span class="token operator">></span><span class="token punctuation">{</span><span class="token operator">&amp;</span><span class="token keyword">self</span><span class="token punctuation">.</span>props<span class="token punctuation">.</span>product<span class="token punctuation">.</span>name<span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span>            <span class="token operator">&lt;</span>div<span class="token operator">></span><span class="token punctuation">{</span><span class="token string">"$"</span><span class="token punctuation">}</span><span class="token punctuation">{</span><span class="token operator">&amp;</span><span class="token keyword">self</span><span class="token punctuation">.</span>props<span class="token punctuation">.</span>product<span class="token punctuation">.</span>price<span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span>            <span class="token operator">&lt;</span>button onclick<span class="token operator">=</span>onclick<span class="token operator">></span><span class="token punctuation">{</span><span class="token string">"Add To Cart"</span><span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>button<span class="token operator">></span>          <span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span>        <span class="token punctuation">}</span>    <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// src/components/mod.rs</span><span class="token keyword">mod</span> product_card<span class="token punctuation">;</span><span class="token keyword">pub</span> <span class="token keyword">use</span> product_card<span class="token punctuation">:</span><span class="token punctuation">:</span>ProductCard<span class="token punctuation">;</span></code></pre><pre class=" language-diff"><code class="language-diff">  // src/lib.rs  mod api;<span class="token inserted">+ mod components;</span>  mod pages;  mod types;  // No changes</code></pre><pre class=" language-diff"><code class="language-diff">  // src/pages/home.rs  use crate::api;<span class="token inserted">+ use crate::components::ProductCard;</span>  use crate::types::{CartProduct, Product};  use anyhow::Error;  use yew::format::Json;  use yew::prelude::*;  use yew::services::fetch::FetchTask;  // No changes  impl Component for Home {      // No changes      fn view(&amp;self) -> Html {          let products: Vec&lt;Html> = self              .state              .products              .iter()              .map(|product: &amp;Product| {                  let product_id = product.id;                  html! {<span class="token deleted">-                   &lt;div></span><span class="token deleted">-                     &lt;img src={&amp;product.image}/></span><span class="token deleted">-                     &lt;div>{&amp;product.name}&lt;/div></span><span class="token deleted">-                     &lt;div>{"$"}{&amp;product.price}&lt;/div></span><span class="token deleted">-                     &lt;button onclick=self.link.callback(move |_| Msg::AddToCart(product_id))>{"Add To Cart"}&lt;/button></span><span class="token deleted">-                   &lt;/div></span><span class="token inserted">+                   &lt;ProductCard product={product} on_add_to_cart=self.link.callback(move |_| Msg::AddToCart(product_id))/></span>                  }              })              .collect();          // No changes      }  }</code></pre><p>Pretty straightforward, except for <code>Properties</code>, <code>Callback</code> and <code>reform</code>.</p><ul><li><code>Properties</code> - As mentioned in the beginning of the post, “Properties” or “Props” are input to a component. If you think of components as functions, then Props are the function arguments.</li><li>For the <code>ProductCard</code> component, we’re passing the <code>Product</code> struct as well as a <code>on_add_to_cart</code> callback. This component doesn’t hold any state, so when user clicks on the “Add To Cart” button, this component calls the parent component to update the <code>cart_products</code> state. This callback is represented using the <code>Callback&lt;T&gt;</code> type and to call this from child component, we either use <code>emit</code> or <code>reform</code> methods on the callback.</li></ul><p><img src="/images/2020-rust-wasm-yew-single-page-application/rust-wasm-yew-single-page-application-4.png" alt=""></p><h2 id="Styling"><a href="#Styling" class="headerlink" title="Styling"></a>Styling</h2><p>The UI looks barebones as we haven’t added any styles.</p><p><img src="/images/2020-rust-wasm-yew-single-page-application/image-3.png" alt=""></p><p>We can either use the <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Global_attributes/class" target="_blank" rel="noopener">class attribute</a> or <a href="https://developer.mozilla.org/en-US/docs/Web/API/ElementCSSInlineStyle/style" target="_blank" rel="noopener">inline styles</a> with Yew. Let’s add some styles so the UI looks good.</p><p>Let’s create a new CSS file <code>static/styles.css</code>, add it to <code>static/index.html</code> and then we can start using the classes in our components.</p><pre class=" language-diff"><code class="language-diff">  // src/pages/home.rs  html! {    &lt;div><span class="token deleted">-     &lt;span>{format!("Cart Value: {:.2}", cart_value)}&lt;/span></span><span class="token deleted">-     &lt;span>{products}&lt;/span></span><span class="token inserted">+     &lt;div class="navbar"></span><span class="token inserted">+         &lt;div class="navbar_title">{"RustMart"}&lt;/div></span><span class="token inserted">+         &lt;div class="navbar_cart_value">{format!("${:.2}", cart_value)}&lt;/div></span><span class="token inserted">+     &lt;/div></span><span class="token inserted">+     &lt;div class="product_card_list">{products}&lt;/div></span>    &lt;/div>  }</code></pre><pre class=" language-diff"><code class="language-diff">  // src/components/product_card.rs  html! {<span class="token deleted">-   &lt;div></span><span class="token deleted">-     &lt;img src={&amp;self.props.product.image}/></span><span class="token deleted">-     &lt;div>{&amp;self.props.product.name}&lt;/div></span><span class="token deleted">-     &lt;div>{"$"}{&amp;self.props.product.price}&lt;/div></span><span class="token deleted">-     &lt;button onclick=onclick>{"Add To Cart"}&lt;/button></span><span class="token deleted">-   &lt;/div></span><span class="token inserted">+   &lt;div class="product_card_container"></span><span class="token inserted">+     &lt;img class="product_card_image" src={&amp;self.props.product.image}/></span><span class="token inserted">+     &lt;div class="product_card_name">{&amp;self.props.product.name}&lt;/div></span><span class="token inserted">+     &lt;div class="product_card_price">{"$"}{&amp;self.props.product.price}&lt;/div></span><span class="token inserted">+     &lt;button class="product_atc_button" onclick=onclick>{"Add To Cart"}&lt;/button></span><span class="token inserted">+   &lt;/div></span>  }</code></pre><p>After adding the styles and a few more products, here’s how the UI looks like:</p><p><img src="/images/2020-rust-wasm-yew-single-page-application/image-4.png" alt=""></p><p>CSS changes are outside the scope of this post, please refer to the <a href="https://github.com/sheshbabu/rustmart-yew-example" target="_blank" rel="noopener">GitHub repo</a>.</p><h2 id="Routing"><a href="#Routing" class="headerlink" title="Routing"></a>Routing</h2><p>In server rendered pages (Jinja, ERB, JSP etc), each page the user sees is mapped to a different template file. For example, when the user navigates to “/login”, it’s rendered in server using “login.html” and when the user goes to “/settings”, it’s rendered using “settings.html”. Using unique urls for different UI pages is also useful for bookmarking and sharing.</p><p>Since SPAs only have one html page (the “Single Page” in SPA), we should be able to replicate the above behavior. This is done using a <code>Router</code>. A Router maps different url paths (with query params, fragments etc) to different page components and helps in navigating between multiple pages without reloading.</p><p>For our application, we’ll be using this mapping:</p><pre class=" language-diff"><code class="language-diff">/            => HomePage/product/:id => ProductDetailPage</code></pre><p>Let’s install <code>yew-router</code>:</p><pre class=" language-diff"><code class="language-diff">  [package]  name = "rustmart"  version = "0.1.0"  authors = ["sheshbabu &lt;sheshbabu@gmail.com>"]  edition = "2018"  [lib]  crate-type = ["cdylib", "rlib"]  [dependencies]  yew = "0.17"<span class="token inserted">+ yew-router = "0.14.0"</span>  wasm-bindgen = "0.2"  log = "0.4.6"  wasm-logger = "0.2.0"  anyhow = "1.0.32"  serde = { version = "1.0", features = ["derive"] }</code></pre><p>Let’s add the routes in a dedicated file so it’s easier to see all available routes at a glance:</p><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// src/route.rs</span><span class="token keyword">use</span> yew_router<span class="token punctuation">:</span><span class="token punctuation">:</span>prelude<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token operator">*</span><span class="token punctuation">;</span><span class="token attribute attr-name">#[derive(Switch, Debug, Clone)]</span><span class="token keyword">pub</span> <span class="token keyword">enum</span> Route <span class="token punctuation">{</span>    #<span class="token punctuation">[</span>to <span class="token operator">=</span> <span class="token string">"/"</span><span class="token punctuation">]</span>    HomePage<span class="token punctuation">,</span><span class="token punctuation">}</span></code></pre><p>For the time being, it only has one route. We’ll add more later.</p><p>Let’s create a new file called <code>src/app.rs</code> to replace <code>HomePage</code> as the new root component:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">use</span> yew<span class="token punctuation">:</span><span class="token punctuation">:</span>prelude<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token operator">*</span><span class="token punctuation">;</span><span class="token keyword">use</span> yew_router<span class="token punctuation">:</span><span class="token punctuation">:</span>prelude<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token operator">*</span><span class="token punctuation">;</span><span class="token keyword">use</span> <span class="token keyword">crate</span><span class="token punctuation">:</span><span class="token punctuation">:</span>pages<span class="token punctuation">:</span><span class="token punctuation">:</span>Home<span class="token punctuation">;</span><span class="token keyword">use</span> <span class="token keyword">crate</span><span class="token punctuation">:</span><span class="token punctuation">:</span>route<span class="token punctuation">:</span><span class="token punctuation">:</span>Route<span class="token punctuation">;</span><span class="token keyword">pub</span> <span class="token keyword">struct</span> App <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token keyword">impl</span> Component <span class="token keyword">for</span> App <span class="token punctuation">{</span>    <span class="token keyword">type</span> Message <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token keyword">type</span> Properties <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token keyword">fn</span> <span class="token function">create</span><span class="token punctuation">(</span>_<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Properties<span class="token punctuation">,</span> _link<span class="token punctuation">:</span> ComponentLink<span class="token operator">&lt;</span>Self<span class="token operator">></span><span class="token punctuation">)</span> <span class="token punctuation">-></span> Self <span class="token punctuation">{</span>        Self <span class="token punctuation">{</span><span class="token punctuation">}</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">update</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">mut</span> <span class="token keyword">self</span><span class="token punctuation">,</span> _msg<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Message<span class="token punctuation">)</span> <span class="token punctuation">-></span> ShouldRender <span class="token punctuation">{</span>        <span class="token keyword">true</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">change</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">mut</span> <span class="token keyword">self</span><span class="token punctuation">,</span> _<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Properties<span class="token punctuation">)</span> <span class="token punctuation">-></span> ShouldRender <span class="token punctuation">{</span>        <span class="token keyword">false</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">view</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">self</span><span class="token punctuation">)</span> <span class="token punctuation">-></span> Html <span class="token punctuation">{</span>        <span class="token keyword">let</span> render <span class="token operator">=</span> Router<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">render</span><span class="token punctuation">(</span><span class="token operator">|</span>switch<span class="token punctuation">:</span> Route<span class="token operator">|</span> <span class="token keyword">match</span> switch <span class="token punctuation">{</span>            Route<span class="token punctuation">:</span><span class="token punctuation">:</span>HomePage <span class="token operator">=</span><span class="token operator">></span> <span class="token macro-rules function">html!</span> <span class="token punctuation">{</span><span class="token operator">&lt;</span>Home<span class="token operator">/</span><span class="token operator">></span><span class="token punctuation">}</span><span class="token punctuation">,</span>        <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>        <span class="token macro-rules function">html!</span> <span class="token punctuation">{</span>            <span class="token operator">&lt;</span>Router<span class="token operator">&lt;</span>Route<span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">></span> render<span class="token operator">=</span>render<span class="token operator">/</span><span class="token operator">></span>        <span class="token punctuation">}</span>    <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><p>Let’s make the corresponding change in <code>lib.rs</code>:</p><pre class=" language-diff"><code class="language-diff">  mod api;<span class="token inserted">+ mod app;</span>  mod components;  mod pages;<span class="token inserted">+ mod route;</span>  mod types;<span class="token deleted">- use pages::Home;</span>  use wasm_bindgen::prelude::*;  use yew::prelude::*;  #[wasm_bindgen(start)]  pub fn run_app() {      wasm_logger::init(wasm_logger::Config::default());<span class="token deleted">-     App::&lt;Home>::new().mount_to_body();</span><span class="token inserted">+     App::&lt;app::App>::new().mount_to_body();</span>  }</code></pre><p>This is how our component hierarchy looks like so far:</p><p><img src="/images/2020-rust-wasm-yew-single-page-application/rust-wasm-yew-single-page-application-5.png" alt=""></p><h2 id="ProductDetailPage"><a href="#ProductDetailPage" class="headerlink" title="ProductDetailPage"></a>ProductDetailPage</h2><p>Now that we have a router in place, let’s use it to navigate from one page to another. Since it’s a SPA, we should avoid page reload while navigating.</p><p>Let’s add a route for ProductDetailPage under <code>/product/:id</code>. When the user clicks on a <code>ProductCard</code>, it will go to its detail page with the <code>id</code> in the route passed as a Prop.</p><pre class=" language-diff"><code class="language-diff">  // src/route.rs  use yew_router::prelude::*;  #[derive(Switch, Debug, Clone)]  pub enum Route {<span class="token inserted">+     #[to = "/product/{id}"]</span><span class="token inserted">+     ProductDetail(i32),</span>      #[to = "/"]      HomePage,  }</code></pre><p>Note that the order of the routes above determines which page gets rendered first. For example, the url <code>/product/2</code> matches both <code>/product/{id}</code> and <code>/</code> but since we wrote <code>/product/{id}</code> first, the <code>ProductDetail</code> page gets rendered instead of <code>Home</code>.</p><p>Add this route to <code>app.rs</code>:</p><pre class=" language-diff"><code class="language-diff">  use yew::prelude::*;  use yew_router::prelude::*;<span class="token deleted">- use crate::pages::{Home};</span><span class="token inserted">+ use crate::pages::{Home, ProductDetail};</span>  use crate::route::Route;  pub struct App {}  impl Component for App {      // No changes      fn view(&amp;self) -> Html {          let render = Router::render(|switch: Route| match switch {<span class="token inserted">+             Route::ProductDetail(id) => html! {&lt;ProductDetail id=id/>},</span>              Route::HomePage => html! {&lt;Home/>},          });          html! {              &lt;Router&lt;Route, ()> render=render/>          }      }  }</code></pre><p>Let’s update the <code>ProductCard</code> so clicking on the product image, name or price takes us to this new page:</p><pre class=" language-diff"><code class="language-diff">  // src/components/product_card.rs<span class="token inserted">+ use crate::route::Route;</span>  use crate::types::Product;  use yew::prelude::*;<span class="token inserted">+ use yew_router::components::RouterAnchor;</span>  // No changes  impl Component for ProductCard {      // No changes      fn view(&amp;self) -> Html {<span class="token inserted">+         type Anchor = RouterAnchor&lt;Route>;</span>          let onclick = self.props.on_add_to_cart.reform(|_| ());          html! {              &lt;div class="product_card_container"><span class="token inserted">+                 &lt;Anchor route=Route::ProductDetail(self.props.product.id) classes="product_card_anchor"></span>                      &lt;img class="product_card_image" src={&amp;self.props.product.image}/>                      &lt;div class="product_card_name">{&amp;self.props.product.name}&lt;/div>                      &lt;div class="product_card_price">{"$"}{&amp;self.props.product.price}&lt;/div><span class="token inserted">+                 &lt;/Anchor></span>                  &lt;button class="product_atc_button" onclick=onclick>{"Add To Cart"}&lt;/button>              &lt;/div>          }      }  }</code></pre><p>Notice how we used <code>classes</code> instead of <code>class</code> for <code>Anchor</code>.</p><p>We’ll create files named <code>static/products/1.json</code>, <code>static/products/2.json</code> etc with mock data:</p><pre class=" language-json"><code class="language-json"><span class="token punctuation">{</span>  <span class="token property">"id"</span><span class="token operator">:</span> <span class="token number">1</span><span class="token punctuation">,</span>  <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"Apple"</span><span class="token punctuation">,</span>  <span class="token property">"description"</span><span class="token operator">:</span> <span class="token string">"An apple a day keeps the doctor away"</span><span class="token punctuation">,</span>  <span class="token property">"image"</span><span class="token operator">:</span> <span class="token string">"/products/apple.png"</span><span class="token punctuation">,</span>  <span class="token property">"price"</span><span class="token operator">:</span> <span class="token number">3.65</span><span class="token punctuation">}</span></code></pre><p>Let’s update the <code>api.rs</code> module with the new route:</p><pre class=" language-diff"><code class="language-diff">  use crate::types::Product;  use anyhow::Error;  use yew::callback::Callback;  use yew::format::{Json, Nothing};  use yew::services::fetch::{FetchService, FetchTask, Request, Response};  pub type FetchResponse&lt;T> = Response&lt;Json&lt;Result&lt;T, Error>>>;  type FetchCallback&lt;T> = Callback&lt;FetchResponse&lt;T>>;  pub fn get_products(callback: FetchCallback&lt;Vec&lt;Product>>) -> FetchTask {      let req = Request::get("/products/products.json")          .body(Nothing)          .unwrap();      FetchService::fetch(req, callback).unwrap()  }<span class="token inserted">+ pub fn get_product(id: i32, callback: FetchCallback&lt;Product>) -> FetchTask {</span><span class="token inserted">+     let req = Request::get(format!("/products/{}.json", id))</span><span class="token inserted">+         .body(Nothing)</span><span class="token inserted">+         .unwrap();</span><span class="token inserted">+</span><span class="token inserted">+     FetchService::fetch(req, callback).unwrap()</span><span class="token inserted">+ }</span></code></pre><p>Finally, here’s the <code>ProductDetail</code> page component:</p><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// src/pages/product_detail.rs</span><span class="token keyword">use</span> <span class="token keyword">crate</span><span class="token punctuation">:</span><span class="token punctuation">:</span>api<span class="token punctuation">;</span><span class="token keyword">use</span> <span class="token keyword">crate</span><span class="token punctuation">:</span><span class="token punctuation">:</span>types<span class="token punctuation">:</span><span class="token punctuation">:</span>Product<span class="token punctuation">;</span><span class="token keyword">use</span> anyhow<span class="token punctuation">:</span><span class="token punctuation">:</span>Error<span class="token punctuation">;</span><span class="token keyword">use</span> yew<span class="token punctuation">:</span><span class="token punctuation">:</span>format<span class="token punctuation">:</span><span class="token punctuation">:</span>Json<span class="token punctuation">;</span><span class="token keyword">use</span> yew<span class="token punctuation">:</span><span class="token punctuation">:</span>prelude<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token operator">*</span><span class="token punctuation">;</span><span class="token keyword">use</span> yew<span class="token punctuation">:</span><span class="token punctuation">:</span>services<span class="token punctuation">:</span><span class="token punctuation">:</span>fetch<span class="token punctuation">:</span><span class="token punctuation">:</span>FetchTask<span class="token punctuation">;</span><span class="token keyword">struct</span> State <span class="token punctuation">{</span>    product<span class="token punctuation">:</span> Option<span class="token operator">&lt;</span>Product<span class="token operator">></span><span class="token punctuation">,</span>    get_product_error<span class="token punctuation">:</span> Option<span class="token operator">&lt;</span>Error<span class="token operator">></span><span class="token punctuation">,</span>    get_product_loaded<span class="token punctuation">:</span> bool<span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token keyword">pub</span> <span class="token keyword">struct</span> ProductDetail <span class="token punctuation">{</span>    props<span class="token punctuation">:</span> Props<span class="token punctuation">,</span>    state<span class="token punctuation">:</span> State<span class="token punctuation">,</span>    link<span class="token punctuation">:</span> ComponentLink<span class="token operator">&lt;</span>Self<span class="token operator">></span><span class="token punctuation">,</span>    task<span class="token punctuation">:</span> Option<span class="token operator">&lt;</span>FetchTask<span class="token operator">></span><span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token attribute attr-name">#[derive(Properties, Clone)]</span><span class="token keyword">pub</span> <span class="token keyword">struct</span> Props <span class="token punctuation">{</span>    <span class="token keyword">pub</span> id<span class="token punctuation">:</span> i32<span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token keyword">pub</span> <span class="token keyword">enum</span> Msg <span class="token punctuation">{</span>    GetProduct<span class="token punctuation">,</span>    <span class="token function">GetProductSuccess</span><span class="token punctuation">(</span>Product<span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token function">GetProductError</span><span class="token punctuation">(</span>Error<span class="token punctuation">)</span><span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token keyword">impl</span> Component <span class="token keyword">for</span> ProductDetail <span class="token punctuation">{</span>    <span class="token keyword">type</span> Message <span class="token operator">=</span> Msg<span class="token punctuation">;</span>    <span class="token keyword">type</span> Properties <span class="token operator">=</span> Props<span class="token punctuation">;</span>    <span class="token keyword">fn</span> <span class="token function">create</span><span class="token punctuation">(</span>props<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Properties<span class="token punctuation">,</span> link<span class="token punctuation">:</span> ComponentLink<span class="token operator">&lt;</span>Self<span class="token operator">></span><span class="token punctuation">)</span> <span class="token punctuation">-></span> Self <span class="token punctuation">{</span>        link<span class="token punctuation">.</span><span class="token function">send_message</span><span class="token punctuation">(</span>Msg<span class="token punctuation">:</span><span class="token punctuation">:</span>GetProduct<span class="token punctuation">)</span><span class="token punctuation">;</span>        Self <span class="token punctuation">{</span>            props<span class="token punctuation">,</span>            state<span class="token punctuation">:</span> State <span class="token punctuation">{</span>                product<span class="token punctuation">:</span> None<span class="token punctuation">,</span>                get_product_error<span class="token punctuation">:</span> None<span class="token punctuation">,</span>                get_product_loaded<span class="token punctuation">:</span> <span class="token keyword">false</span><span class="token punctuation">,</span>            <span class="token punctuation">}</span><span class="token punctuation">,</span>            link<span class="token punctuation">,</span>            task<span class="token punctuation">:</span> None<span class="token punctuation">,</span>        <span class="token punctuation">}</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">update</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">mut</span> <span class="token keyword">self</span><span class="token punctuation">,</span> message<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Message<span class="token punctuation">)</span> <span class="token punctuation">-></span> ShouldRender <span class="token punctuation">{</span>        <span class="token keyword">match</span> message <span class="token punctuation">{</span>            Msg<span class="token punctuation">:</span><span class="token punctuation">:</span>GetProduct <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span>                <span class="token keyword">let</span> handler <span class="token operator">=</span> <span class="token keyword">self</span>                    <span class="token punctuation">.</span>link                    <span class="token punctuation">.</span><span class="token function">callback</span><span class="token punctuation">(</span><span class="token keyword">move</span> <span class="token closure-params"><span class="token punctuation">|</span>response<span class="token punctuation">:</span> api<span class="token punctuation">:</span><span class="token punctuation">:</span>FetchResponse&lt;Product><span class="token punctuation">|</span></span> <span class="token punctuation">{</span>                        <span class="token keyword">let</span> <span class="token punctuation">(</span>_<span class="token punctuation">,</span> <span class="token function">Json</span><span class="token punctuation">(</span>data<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">=</span> response<span class="token punctuation">.</span><span class="token function">into_parts</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>                        <span class="token keyword">match</span> data <span class="token punctuation">{</span>                            <span class="token function">Ok</span><span class="token punctuation">(</span>product<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> Msg<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">GetProductSuccess</span><span class="token punctuation">(</span>product<span class="token punctuation">)</span><span class="token punctuation">,</span>                            <span class="token function">Err</span><span class="token punctuation">(</span>err<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> Msg<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">GetProductError</span><span class="token punctuation">(</span>err<span class="token punctuation">)</span><span class="token punctuation">,</span>                        <span class="token punctuation">}</span>                    <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>                <span class="token keyword">self</span><span class="token punctuation">.</span>task <span class="token operator">=</span> <span class="token function">Some</span><span class="token punctuation">(</span>api<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">get_product</span><span class="token punctuation">(</span><span class="token keyword">self</span><span class="token punctuation">.</span>props<span class="token punctuation">.</span>id<span class="token punctuation">,</span> handler<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>                <span class="token keyword">true</span>            <span class="token punctuation">}</span>            Msg<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">GetProductSuccess</span><span class="token punctuation">(</span>product<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span>                <span class="token keyword">self</span><span class="token punctuation">.</span>state<span class="token punctuation">.</span>product <span class="token operator">=</span> <span class="token function">Some</span><span class="token punctuation">(</span>product<span class="token punctuation">)</span><span class="token punctuation">;</span>                <span class="token keyword">self</span><span class="token punctuation">.</span>state<span class="token punctuation">.</span>get_product_loaded <span class="token operator">=</span> <span class="token keyword">true</span><span class="token punctuation">;</span>                <span class="token keyword">true</span>            <span class="token punctuation">}</span>            Msg<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">GetProductError</span><span class="token punctuation">(</span>error<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span>                <span class="token keyword">self</span><span class="token punctuation">.</span>state<span class="token punctuation">.</span>get_product_error <span class="token operator">=</span> <span class="token function">Some</span><span class="token punctuation">(</span>error<span class="token punctuation">)</span><span class="token punctuation">;</span>                <span class="token keyword">self</span><span class="token punctuation">.</span>state<span class="token punctuation">.</span>get_product_loaded <span class="token operator">=</span> <span class="token keyword">true</span><span class="token punctuation">;</span>                <span class="token keyword">true</span>            <span class="token punctuation">}</span>        <span class="token punctuation">}</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">change</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">mut</span> <span class="token keyword">self</span><span class="token punctuation">,</span> _<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Properties<span class="token punctuation">)</span> <span class="token punctuation">-></span> ShouldRender <span class="token punctuation">{</span>        <span class="token keyword">false</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">view</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">self</span><span class="token punctuation">)</span> <span class="token punctuation">-></span> Html <span class="token punctuation">{</span>        <span class="token keyword">if</span> <span class="token keyword">let</span> <span class="token function">Some</span><span class="token punctuation">(</span><span class="token keyword">ref</span> product<span class="token punctuation">)</span> <span class="token operator">=</span> <span class="token keyword">self</span><span class="token punctuation">.</span>state<span class="token punctuation">.</span>product <span class="token punctuation">{</span>            <span class="token macro-rules function">html!</span> <span class="token punctuation">{</span>                <span class="token operator">&lt;</span>div class<span class="token operator">=</span><span class="token string">"product_detail_container"</span><span class="token operator">></span>                    <span class="token operator">&lt;</span>img class<span class="token operator">=</span><span class="token string">"product_detail_image"</span> src<span class="token operator">=</span><span class="token punctuation">{</span><span class="token operator">&amp;</span>product<span class="token punctuation">.</span>image<span class="token punctuation">}</span><span class="token operator">/</span><span class="token operator">></span>                    <span class="token operator">&lt;</span>div class<span class="token operator">=</span><span class="token string">"product_card_name"</span><span class="token operator">></span><span class="token punctuation">{</span><span class="token operator">&amp;</span>product<span class="token punctuation">.</span>name<span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span>                    <span class="token operator">&lt;</span>div style<span class="token operator">=</span><span class="token string">"margin: 10px 0; line-height: 24px;"</span><span class="token operator">></span><span class="token punctuation">{</span><span class="token operator">&amp;</span>product<span class="token punctuation">.</span>description<span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span>                    <span class="token operator">&lt;</span>div class<span class="token operator">=</span><span class="token string">"product_card_price"</span><span class="token operator">></span><span class="token punctuation">{</span><span class="token string">"$"</span><span class="token punctuation">}</span><span class="token punctuation">{</span><span class="token operator">&amp;</span>product<span class="token punctuation">.</span>price<span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span>                    <span class="token operator">&lt;</span>button class<span class="token operator">=</span><span class="token string">"product_atc_button"</span><span class="token operator">></span><span class="token punctuation">{</span><span class="token string">"Add To Cart"</span><span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>button<span class="token operator">></span>                <span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span>            <span class="token punctuation">}</span>        <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token keyword">if</span> <span class="token operator">!</span><span class="token keyword">self</span><span class="token punctuation">.</span>state<span class="token punctuation">.</span>get_product_loaded <span class="token punctuation">{</span>            <span class="token macro-rules function">html!</span> <span class="token punctuation">{</span>                <span class="token operator">&lt;</span>div class<span class="token operator">=</span><span class="token string">"loading_spinner_container"</span><span class="token operator">></span>                    <span class="token operator">&lt;</span>div class<span class="token operator">=</span><span class="token string">"loading_spinner"</span><span class="token operator">></span><span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span>                    <span class="token operator">&lt;</span>div class<span class="token operator">=</span><span class="token string">"loading_spinner_text"</span><span class="token operator">></span><span class="token punctuation">{</span><span class="token string">"Loading ..."</span><span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span>                <span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span>            <span class="token punctuation">}</span>        <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>            <span class="token macro-rules function">html!</span> <span class="token punctuation">{</span>                <span class="token operator">&lt;</span>div<span class="token operator">></span>                    <span class="token operator">&lt;</span>span<span class="token operator">></span><span class="token punctuation">{</span><span class="token string">"Error loading product! :("</span><span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>span<span class="token operator">></span>                <span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span>            <span class="token punctuation">}</span>        <span class="token punctuation">}</span>    <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><p>Very similar to the HomePage component. Let’s also add this file to the module tree:</p><pre class=" language-diff"><code class="language-diff">  // src/pages/mod.rs  mod home;<span class="token inserted">+ mod product_detail;</span>  pub use home::Home;<span class="token inserted">+ pub use product_detail::ProductDetail;</span></code></pre><p>This is how it looks like:</p><p><img src="/images/2020-rust-wasm-yew-single-page-application/image-5.png" alt=""></p><p>We can now move between multiple pages without refreshing the page!</p><h2 id="State-Management"><a href="#State-Management" class="headerlink" title="State Management"></a>State Management</h2><p>One thing you might have noticed in the <code>ProductDetail</code> page is that clicking on the “Add To Cart” button doesn’t update the cart. This is because the state that holds the list of products in cart <code>cart_products</code> currently resides inside <code>Home</code> page component:<br><img src="/images/2020-rust-wasm-yew-single-page-application/rust-wasm-yew-single-page-application-6.png" alt=""></p><p>To share state between two components, we can either:</p><ul><li>Hoist the state to a common ancestor</li><li>Move state to global app state</li></ul><p>The <code>App</code> component is a common ancestor to both <code>ProductDetail</code> and <code>Home</code>. We can move the <code>cart_products</code> state there and pass it as props to <code>ProductDetail</code> and <code>Home</code>.</p><p><img src="/images/2020-rust-wasm-yew-single-page-application/rust-wasm-yew-single-page-application-7.png" alt=""></p><p>This works fine for shallow component hierarchies but when you have deep component hierarchy (which is common in larger SPAs), you’ll need to pass this state through multiple layers of components (which might not have use for this prop) to reach the desired node. This is called “Prop Drilling”.</p><p>You can see that <code>cart_products</code> is now passed from <code>App</code> to <code>AddToCart</code> component via <code>ProductDetail</code> and <code>Home</code> even though they have no use for this state. Imagine the same scenario with components many layers deep.</p><p>This is the problem the global state solves. Here’s how it would look like:</p><p><img src="/images/2020-rust-wasm-yew-single-page-application/rust-wasm-yew-single-page-application-8.png" alt=""></p><p>Notice how there’s a direct link between the components that need this state and the global state.</p><p>Unfortunately, Yew doesn’t seem to have a <a href="https://github.com/yewstack/yew/issues/576" target="_blank" rel="noopener">good solution</a> for this. The recommended solution is to use <code>Agents</code> for broadcasting state changes via pubsub. This is something I stay away from as it gets messy fast. I hope in future we see something similar to React’s <a href="https://reactjs.org/docs/context.html" target="_blank" rel="noopener">Context</a>, Redux or Mobx etc.</p><p>Let’s solve our problem by hoisting the state.</p><h2 id="Hoisting-State"><a href="#Hoisting-State" class="headerlink" title="Hoisting State"></a>Hoisting State</h2><p>We’ll be refactoring our code by moving <code>cart_products</code> state to <code>App</code> and extracting <code>Navbar</code> and <code>AtcButton</code> as separate components:</p><p><img src="/images/2020-rust-wasm-yew-single-page-application/rust-wasm-yew-single-page-application-9.png" alt=""></p><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// src/components/navbar.rs</span><span class="token keyword">use</span> <span class="token keyword">crate</span><span class="token punctuation">:</span><span class="token punctuation">:</span>types<span class="token punctuation">:</span><span class="token punctuation">:</span>CartProduct<span class="token punctuation">;</span><span class="token keyword">use</span> yew<span class="token punctuation">:</span><span class="token punctuation">:</span>prelude<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token operator">*</span><span class="token punctuation">;</span><span class="token keyword">pub</span> <span class="token keyword">struct</span> Navbar <span class="token punctuation">{</span>    props<span class="token punctuation">:</span> Props<span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token attribute attr-name">#[derive(Properties, Clone)]</span><span class="token keyword">pub</span> <span class="token keyword">struct</span> Props <span class="token punctuation">{</span>    <span class="token keyword">pub</span> cart_products<span class="token punctuation">:</span> Vec<span class="token operator">&lt;</span>CartProduct<span class="token operator">></span><span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token keyword">impl</span> Component <span class="token keyword">for</span> Navbar <span class="token punctuation">{</span>    <span class="token keyword">type</span> Message <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token keyword">type</span> Properties <span class="token operator">=</span> Props<span class="token punctuation">;</span>    <span class="token keyword">fn</span> <span class="token function">create</span><span class="token punctuation">(</span>props<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Properties<span class="token punctuation">,</span> _link<span class="token punctuation">:</span> ComponentLink<span class="token operator">&lt;</span>Self<span class="token operator">></span><span class="token punctuation">)</span> <span class="token punctuation">-></span> Self <span class="token punctuation">{</span>        Self <span class="token punctuation">{</span> props <span class="token punctuation">}</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">update</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">mut</span> <span class="token keyword">self</span><span class="token punctuation">,</span> _msg<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Message<span class="token punctuation">)</span> <span class="token punctuation">-></span> ShouldRender <span class="token punctuation">{</span>        <span class="token keyword">true</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">change</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">mut</span> <span class="token keyword">self</span><span class="token punctuation">,</span> props<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Properties<span class="token punctuation">)</span> <span class="token punctuation">-></span> ShouldRender <span class="token punctuation">{</span>        <span class="token keyword">self</span><span class="token punctuation">.</span>props <span class="token operator">=</span> props<span class="token punctuation">;</span>        <span class="token keyword">true</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">view</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">self</span><span class="token punctuation">)</span> <span class="token punctuation">-></span> Html <span class="token punctuation">{</span>        <span class="token keyword">let</span> cart_value <span class="token operator">=</span> <span class="token keyword">self</span>            <span class="token punctuation">.</span>props            <span class="token punctuation">.</span>cart_products            <span class="token punctuation">.</span><span class="token function">iter</span><span class="token punctuation">(</span><span class="token punctuation">)</span>            <span class="token punctuation">.</span><span class="token function">fold</span><span class="token punctuation">(</span><span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token operator">|</span>acc<span class="token punctuation">,</span> cp<span class="token operator">|</span> acc <span class="token operator">+</span> <span class="token punctuation">(</span>cp<span class="token punctuation">.</span>quantity <span class="token keyword">as</span> f64 <span class="token operator">*</span> cp<span class="token punctuation">.</span>product<span class="token punctuation">.</span>price<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>        <span class="token macro-rules function">html!</span> <span class="token punctuation">{</span>            <span class="token operator">&lt;</span>div class<span class="token operator">=</span><span class="token string">"navbar"</span><span class="token operator">></span>                <span class="token operator">&lt;</span>div class<span class="token operator">=</span><span class="token string">"navbar_title"</span><span class="token operator">></span><span class="token punctuation">{</span><span class="token string">"RustMart"</span><span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span>              <span class="token operator">&lt;</span>div class<span class="token operator">=</span><span class="token string">"navbar_cart_value"</span><span class="token operator">></span><span class="token punctuation">{</span><span class="token function">format!</span><span class="token punctuation">(</span><span class="token string">"${:.2}"</span><span class="token punctuation">,</span> cart_value<span class="token punctuation">)</span><span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span>            <span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">></span>        <span class="token punctuation">}</span>    <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><p>Notice how we started using the <code>change</code> lifecycle methods in the <code>Navbar</code> component. When the props sent from parent changes, we need to update the props inside the component so the UI re-renders.</p><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// src/components/atc_button.rs</span><span class="token keyword">use</span> <span class="token keyword">crate</span><span class="token punctuation">:</span><span class="token punctuation">:</span>types<span class="token punctuation">:</span><span class="token punctuation">:</span>Product<span class="token punctuation">;</span><span class="token keyword">use</span> yew<span class="token punctuation">:</span><span class="token punctuation">:</span>prelude<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token operator">*</span><span class="token punctuation">;</span><span class="token keyword">pub</span> <span class="token keyword">struct</span> AtcButton <span class="token punctuation">{</span>    props<span class="token punctuation">:</span> Props<span class="token punctuation">,</span>    link<span class="token punctuation">:</span> ComponentLink<span class="token operator">&lt;</span>Self<span class="token operator">></span><span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token attribute attr-name">#[derive(Properties, Clone)]</span><span class="token keyword">pub</span> <span class="token keyword">struct</span> Props <span class="token punctuation">{</span>    <span class="token keyword">pub</span> product<span class="token punctuation">:</span> Product<span class="token punctuation">,</span>    <span class="token keyword">pub</span> on_add_to_cart<span class="token punctuation">:</span> Callback<span class="token operator">&lt;</span>Product<span class="token operator">></span><span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token keyword">pub</span> <span class="token keyword">enum</span> Msg <span class="token punctuation">{</span>    AddToCart<span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token keyword">impl</span> Component <span class="token keyword">for</span> AtcButton <span class="token punctuation">{</span>    <span class="token keyword">type</span> Message <span class="token operator">=</span> Msg<span class="token punctuation">;</span>    <span class="token keyword">type</span> Properties <span class="token operator">=</span> Props<span class="token punctuation">;</span>    <span class="token keyword">fn</span> <span class="token function">create</span><span class="token punctuation">(</span>props<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Properties<span class="token punctuation">,</span> link<span class="token punctuation">:</span> ComponentLink<span class="token operator">&lt;</span>Self<span class="token operator">></span><span class="token punctuation">)</span> <span class="token punctuation">-></span> Self <span class="token punctuation">{</span>        Self <span class="token punctuation">{</span> props<span class="token punctuation">,</span> link <span class="token punctuation">}</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">update</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">mut</span> <span class="token keyword">self</span><span class="token punctuation">,</span> msg<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Message<span class="token punctuation">)</span> <span class="token punctuation">-></span> ShouldRender <span class="token punctuation">{</span>        <span class="token keyword">match</span> msg <span class="token punctuation">{</span>            Msg<span class="token punctuation">:</span><span class="token punctuation">:</span>AddToCart <span class="token operator">=</span><span class="token operator">></span> <span class="token keyword">self</span><span class="token punctuation">.</span>props<span class="token punctuation">.</span>on_add_to_cart<span class="token punctuation">.</span><span class="token function">emit</span><span class="token punctuation">(</span><span class="token keyword">self</span><span class="token punctuation">.</span>props<span class="token punctuation">.</span>product<span class="token punctuation">.</span><span class="token function">clone</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">,</span>        <span class="token punctuation">}</span>        <span class="token keyword">true</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">change</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">mut</span> <span class="token keyword">self</span><span class="token punctuation">,</span> props<span class="token punctuation">:</span> Self<span class="token punctuation">:</span><span class="token punctuation">:</span>Properties<span class="token punctuation">)</span> <span class="token punctuation">-></span> ShouldRender <span class="token punctuation">{</span>        <span class="token keyword">self</span><span class="token punctuation">.</span>props <span class="token operator">=</span> props<span class="token punctuation">;</span>        <span class="token keyword">true</span>    <span class="token punctuation">}</span>    <span class="token keyword">fn</span> <span class="token function">view</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">self</span><span class="token punctuation">)</span> <span class="token punctuation">-></span> Html <span class="token punctuation">{</span>        <span class="token keyword">let</span> onclick <span class="token operator">=</span> <span class="token keyword">self</span><span class="token punctuation">.</span>link<span class="token punctuation">.</span><span class="token function">callback</span><span class="token punctuation">(</span><span class="token operator">|</span>_<span class="token operator">|</span> Msg<span class="token punctuation">:</span><span class="token punctuation">:</span>AddToCart<span class="token punctuation">)</span><span class="token punctuation">;</span>        <span class="token macro-rules function">html!</span> <span class="token punctuation">{</span>          <span class="token operator">&lt;</span>button class<span class="token operator">=</span><span class="token string">"product_atc_button"</span> onclick<span class="token operator">=</span>onclick<span class="token operator">></span><span class="token punctuation">{</span><span class="token string">"Add To Cart"</span><span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>button<span class="token operator">></span>        <span class="token punctuation">}</span>    <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><pre class=" language-diff"><code class="language-diff">  // src/components/mod.rs<span class="token inserted">+ mod atc_button;</span><span class="token inserted">+ mod navbar;</span>  mod product_card;<span class="token inserted">+ pub use atc_button::AtcButton;</span><span class="token inserted">+ pub use navbar::Navbar;</span>  pub use product_card::ProductCard;</code></pre><p>Use the new <code>AtcButton</code> in <code>ProductCard</code> and <code>ProductDetail</code>:</p><pre class=" language-diff"><code class="language-diff">  // src/components/product_card.rs<span class="token inserted">+ use crate::components::AtcButton;</span>  use crate::route::Route;  use crate::types::Product;  use yew::prelude::*;  use yew_router::components::RouterAnchor;  pub struct ProductCard {      props: Props,  }  #[derive(Properties, Clone)]  pub struct Props {      pub product: Product,<span class="token deleted">-     pub on_add_to_cart: Callback&lt;()>,</span><span class="token inserted">+     pub on_add_to_cart: Callback&lt;Product>,</span>  }  impl Component for ProductCard {      // No changes      fn view(&amp;self) -> Html {          type Anchor = RouterAnchor&lt;Route>;<span class="token deleted">-         let onclick = self.props.on_add_to_cart.reform(|_| ());</span>          html! {              &lt;div class="product_card_container">                  &lt;Anchor route=Route::ProductDetail(self.props.product.id) classes="product_card_anchor">                      &lt;img class="product_card_image" src={&amp;self.props.product.image}/>                      &lt;div class="product_card_name">{&amp;self.props.product.name}&lt;/div>                      &lt;div class="product_card_price">{"$"}{&amp;self.props.product.price}&lt;/div>                  &lt;/Anchor><span class="token deleted">-                 &lt;button class="product_atc_button" onclick=onclick>{"Add To Cart"}&lt;/button></span><span class="token inserted">+                 &lt;AtcButton product=self.props.product.clone() on_add_to_cart=self.props.on_add_to_cart.clone() /></span>              &lt;/div>          }      }  }</code></pre><pre class=" language-diff"><code class="language-diff">  // src/pages/product_detail.rs  use crate::api;<span class="token inserted">+ use crate::components::AtcButton;</span>  use crate::types::Product;  use anyhow::Error;  use yew::format::Json;  use yew::prelude::*;  use yew::services::fetch::FetchTask;  // No changes  #[derive(Properties, Clone)]  pub struct Props {      pub id: i32,<span class="token inserted">+     pub on_add_to_cart: Callback&lt;Product>,</span>  }  impl Component for ProductDetail {      // No changes      fn view(&amp;self) -> Html {          if let Some(ref product) = self.state.product {              html! {                  &lt;div class="product_detail_container">                      &lt;img class="product_detail_image" src={&amp;product.image}/>                      &lt;div class="product_card_name">{&amp;product.name}&lt;/div>                      &lt;div style="margin: 10px 0; line-height: 24px;">{&amp;product.description}&lt;/div>                      &lt;div class="product_card_price">{"$"}{&amp;product.price}&lt;/div><span class="token deleted">-                     &lt;button class="product_atc_button">{"Add To Cart"}&lt;/button></span><span class="token inserted">+                     &lt;AtcButton product=product.clone() on_add_to_cart=self.props.on_add_to_cart.clone() /></span>                  &lt;/div>              }          }          // No changes      }  }</code></pre><p>Finally, move the <code>cart_products</code> state from <code>Home</code> to <code>App</code>:</p><pre class=" language-diff"><code class="language-diff">  // src/app.rs<span class="token inserted">+ use crate::components::Navbar;</span><span class="token inserted">+ use crate::types::{CartProduct, Product};</span>  use yew::prelude::*;  use yew_router::prelude::*;  use crate::pages::{Home, ProductDetail};  use crate::route::Route;<span class="token inserted">+ struct State {</span><span class="token inserted">+     cart_products: Vec&lt;CartProduct>,</span><span class="token inserted">+ }</span><span class="token deleted">- pub struct App {}</span><span class="token inserted">+ pub struct App {</span><span class="token inserted">+     state: State,</span><span class="token inserted">+     link: ComponentLink&lt;Self>,</span><span class="token inserted">+ }</span><span class="token inserted">+ pub enum Msg {</span><span class="token inserted">+     AddToCart(Product),</span><span class="token inserted">+ }</span>  impl Component for App {<span class="token deleted">-     type Message = ();</span><span class="token inserted">+     type Message = Msg;</span>      type Properties = ();<span class="token deleted">-     fn create(_: Self::Properties, _link: ComponentLink&lt;Self>) -> Self {</span><span class="token inserted">+     fn create(_: Self::Properties, link: ComponentLink&lt;Self>) -> Self {</span><span class="token inserted">+         let cart_products = vec![];</span><span class="token deleted">-         Self {}</span><span class="token inserted">+         Self {</span><span class="token inserted">+             state: State { cart_products },</span><span class="token inserted">+             link,</span><span class="token inserted">+         }</span>      }<span class="token deleted">-     fn update(&amp;mut self, _msg: Self::Message) -> ShouldRender {</span><span class="token inserted">+     fn update(&amp;mut self, message: Self::Message) -> ShouldRender {</span><span class="token inserted">+         match message {</span><span class="token inserted">+             Msg::AddToCart(product) => {</span><span class="token inserted">+                 let cart_product = self</span><span class="token inserted">+                     .state</span><span class="token inserted">+                     .cart_products</span><span class="token inserted">+                     .iter_mut()</span><span class="token inserted">+                     .find(|cp: &amp;&amp;mut CartProduct| cp.product.id == product.id);</span><span class="token inserted">+                 if let Some(cp) = cart_product {</span><span class="token inserted">+                     cp.quantity += 1;</span><span class="token inserted">+                 } else {</span><span class="token inserted">+                     self.state.cart_products.push(CartProduct {</span><span class="token inserted">+                         product: product.clone(),</span><span class="token inserted">+                         quantity: 1,</span><span class="token inserted">+                     })</span><span class="token inserted">+                 }</span><span class="token inserted">+                 true</span><span class="token inserted">+             }</span><span class="token inserted">+         }</span><span class="token deleted">-         true</span>      }      fn change(&amp;mut self, _: Self::Properties) -> ShouldRender {          false      }      fn view(&amp;self) -> Html {<span class="token inserted">+         let handle_add_to_cart = self</span><span class="token inserted">+             .link</span><span class="token inserted">+             .callback(|product: Product| Msg::AddToCart(product));</span><span class="token inserted">+         let cart_products = self.state.cart_products.clone();</span><span class="token deleted">-         let render = Router::render(|switch: Route| match switch {</span><span class="token deleted">-           Route::ProductDetail(id) => html! {&lt;ProductDetail id=id/>},</span><span class="token deleted">-           Route::HomePage => html! {&lt;Home/>},</span><span class="token inserted">+         let render = Router::render(move |switch: Route| match switch {</span><span class="token inserted">+             Route::ProductDetail(id) => {</span><span class="token inserted">+                 html! {&lt;ProductDetail id=id on_add_to_cart=handle_add_to_cart.clone() />}</span><span class="token inserted">+             }</span><span class="token inserted">+             Route::HomePage => {</span><span class="token inserted">+                 html! {&lt;Home cart_products=cart_products.clone() on_add_to_cart=handle_add_to_cart.clone()/>}</span><span class="token inserted">+             }</span>          });          html! {<span class="token inserted">+             &lt;></span><span class="token inserted">+                 &lt;Navbar cart_products=self.state.cart_products.clone() /></span>                  &lt;Router&lt;Route, ()> render=render/><span class="token inserted">+             &lt;/></span>          }      }  }</code></pre><pre class=" language-diff"><code class="language-diff">  // src/pages/home.rs  // No changes  struct State {      products: Vec&lt;Product>,<span class="token deleted">-     cart_products: Vec&lt;CartProduct>,</span>      get_products_error: Option&lt;Error>,      get_products_loaded: bool,  }<span class="token inserted">+ #[derive(Properties, Clone)]</span><span class="token inserted">+ pub struct Props {</span><span class="token inserted">+     pub cart_products: Vec&lt;CartProduct>,</span><span class="token inserted">+     pub on_add_to_cart: Callback&lt;Product>,</span><span class="token inserted">+ }</span>  pub struct Home {<span class="token inserted">+     props: Props,</span>      state: State,      link: ComponentLink&lt;Self>,      task: Option&lt;FetchTask>,  }  pub enum Msg {<span class="token deleted">-     AddToCart(i32),</span>      GetProducts,      GetProductsSuccess(Vec&lt;Product>),      GetProductsError(Error),  }  impl Component for Home {      type Message = Msg;<span class="token deleted">-     type Properties = ();</span><span class="token inserted">+     type Properties = Props;</span><span class="token deleted">-     fn create(_: Self::Properties, link: ComponentLink&lt;Self>) -> Self {</span><span class="token inserted">+     fn create(props: Self::Properties, link: ComponentLink&lt;Self>) -> Self {</span>          let products = vec![];<span class="token deleted">-         let cart_products = vec![];</span>          link.send_message(Msg::GetProducts);          Self {              props,              state: State {                  products,<span class="token deleted">-                 cart_products,</span>                  get_products_error: None,                  get_products_loaded: false,              },              link,              task: None,          }      }      fn update(&amp;mut self, message: Self::Message) -> ShouldRender {          match message {              Msg::GetProducts => {                  self.state.get_products_loaded = false;                  let handler =                      self.link                          .callback(move |response: api::FetchResponse&lt;Vec&lt;Product>>| {                              let (_, Json(data)) = response.into_parts();                              match data {                                  Ok(products) => Msg::GetProductsSuccess(products),                                  Err(err) => Msg::GetProductsError(err),                              }                          });                  self.task = Some(api::get_products(handler));                  true              }              Msg::GetProductsSuccess(products) => {                  self.state.products = products;                  self.state.get_products_loaded = true;                  true              }              Msg::GetProductsError(error) => {                  self.state.get_products_error = Some(error);                  self.state.get_products_loaded = true;                  true              }<span class="token deleted">-             Msg::AddToCart(product_id) => {</span><span class="token deleted">-                 let product = self</span><span class="token deleted">-                     .state</span><span class="token deleted">-                     .products</span><span class="token deleted">-                     .iter()</span><span class="token deleted">-                     .find(|p: &amp;&amp;Product| p.id == product_id)</span><span class="token deleted">-                     .unwrap();</span><span class="token deleted">-                 let cart_product = self</span><span class="token deleted">-                     .state</span><span class="token deleted">-                     .cart_products</span><span class="token deleted">-                     .iter_mut()</span><span class="token deleted">-                     .find(|cp: &amp;&amp;mut CartProduct| cp.product.id == product_id);</span><span class="token deleted">-                 if let Some(cp) = cart_product {</span><span class="token deleted">-                     cp.quantity += 1;</span><span class="token deleted">-                 } else {</span><span class="token deleted">-                     self.state.cart_products.push(CartProduct {</span><span class="token deleted">-                         product: product.clone(),</span><span class="token deleted">-                         quantity: 1,</span><span class="token deleted">-                     })</span><span class="token deleted">-                 }</span><span class="token deleted">-                 true</span><span class="token deleted">-             }</span>          }      }<span class="token deleted">-     fn change(&amp;mut self, _: Self::Properties) -> ShouldRender {</span><span class="token inserted">+     fn change(&amp;mut self, props: Self::Properties) -> ShouldRender {</span><span class="token inserted">+         self.props = props;</span>          true      }      fn view(&amp;self) -> Html {          let products: Vec&lt;Html> = self              .state              .products              .iter()              .map(|product: &amp;Product| {<span class="token deleted">-                 let product_id = product.id;</span>                  html! {<span class="token deleted">-                   &lt;ProductCard product={product} on_add_to_cart=self.link.callback(move |_| Msg::AddToCart(product_id))/></span><span class="token inserted">+                   &lt;ProductCard product={product} on_add_to_cart=self.props.on_add_to_cart.clone()/></span>                  }              })              .collect();<span class="token deleted">-        let cart_value = self</span><span class="token deleted">-            .state</span><span class="token deleted">-            .cart_products</span><span class="token deleted">-            .iter()</span><span class="token deleted">-            .fold(0.0, |acc, cp| acc + (cp.quantity as f64 * cp.product.price));</span>          if !self.state.get_products_loaded {              // No changes          } else if let Some(_) = self.state.get_products_error {              // No changes          } else {              html! {<span class="token deleted">-               &lt;div></span><span class="token deleted">-                 &lt;div class="navbar"></span><span class="token deleted">-                     &lt;div class="navbar_title">{"RustMart"}&lt;/div></span><span class="token deleted">-                     &lt;div class="navbar_cart_value">{format!("${:.2}", cart_value)}&lt;/div></span><span class="token deleted">-                 &lt;/div></span>                  &lt;div class="product_card_list">{products}&lt;/div><span class="token deleted">-               &lt;/div></span>              }          }      }  }</code></pre><p>Now we can finally add to cart from <code>ProductDetail</code> page and we can also see the navbar in all pages</p><p><img src="/images/2020-rust-wasm-yew-single-page-application/image-6.png" alt=""><br><img src="/images/2020-rust-wasm-yew-single-page-application/image-4.png" alt=""></p><p>We’ve successfully built a SPA fully in Rust!</p><p>I’ve hosted the demo <a href="https://rustmart-yew.netlify.app" target="_blank" rel="noopener">here</a> and the code is in this <a href="https://github.com/sheshbabu/rustmart-yew-example" target="_blank" rel="noopener">GitHub repo</a>. If you have questions or suggestions, please contact me at sheshbabu [at] gmail.com.</p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>The Yew community has done a good job designing abstractions like <code>html!</code>, <code>Component</code> etc so someone like me who’s familiar with React can immediately start being productive. It definitely has some rough edges like FetchTask, lack of <a href="https://redux.js.org/introduction/motivation" target="_blank" rel="noopener"><em>predictable</em></a> state management and the documentation is sparse, but has potential to become a good alternative to React, Vue etc once these issues are fixed.</p><p>Thanks for reading! Feel free to follow me in <a href="https://twitter.com/sheshbabu" target="_blank" rel="noopener">Twitter</a> for more posts like this :)</p>]]></content>
    
    <summary type="html">
    
      Easy to follow guide to building SPAs using Rust, WebAssembly and Yew
    
    </summary>
    
    
      <category term="Rust" scheme="https://www.sheshbabu.com/tags/Rust/"/>
    
  </entry>
  
  <entry>
    <title>Beginner&#39;s guide to Error Handling in Rust</title>
    <link href="https://www.sheshbabu.com/posts/rust-error-handling/"/>
    <id>https://www.sheshbabu.com/posts/rust-error-handling/</id>
    <published>2020-08-02T13:00:00.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<p>Error handling in Rust is very different if you’re coming from other languages. In languages like Java, JS, Python etc, you usually <code>throw</code> exceptions and <code>return</code> successful values. In Rust, you return something called a <code>Result</code>.</p><p>The <code>Result&lt;T, E&gt;</code> type is an <a href="https://doc.rust-lang.org/book/ch06-01-defining-an-enum.html" target="_blank" rel="noopener">enum</a> that has two variants - <code>Ok(T)</code> for successful value or <code>Err(E)</code> for error value:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">enum</span> Result<span class="token operator">&lt;</span>T<span class="token punctuation">,</span> E<span class="token operator">></span> <span class="token punctuation">{</span>   <span class="token function">Ok</span><span class="token punctuation">(</span>T<span class="token punctuation">)</span><span class="token punctuation">,</span>   <span class="token function">Err</span><span class="token punctuation">(</span>E<span class="token punctuation">)</span><span class="token punctuation">,</span><span class="token punctuation">}</span></code></pre><p>Returning errors instead of throwing them is a paradigm shift in error handling. If you’re new to Rust, there will be some friction initially as it requires you to reason about errors in a different way.</p><p>In this post, I’ll go through some common error handling patterns so you gradually become familiar with how things are done in Rust:</p><ul><li>Ignore the error</li><li>Terminate the program</li><li>Use a fallback value</li><li>Bubble up the error</li><li>Bubble up multiple errors</li><li>Match boxed errors</li><li>Libraries vs Applications</li><li>Create custom errors</li><li>Bubble up custom errors</li><li>Match custom errors</li></ul><h2 id="Ignore-the-error"><a href="#Ignore-the-error" class="headerlink" title="Ignore the error"></a>Ignore the error</h2><p>Let’s start with the simplest scenario where we just ignore the error. This sounds careless but has a couple of legitimate use cases:</p><ul><li>We’re prototyping our code and don’t want to spend time on error handling.</li><li>We’re confident that the error won’t occur.</li></ul><p>Let’s say that we’re reading a file which we’re pretty sure would be present:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">use</span> std<span class="token punctuation">:</span><span class="token punctuation">:</span>fs<span class="token punctuation">;</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> content <span class="token operator">=</span> fs<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">read_to_string</span><span class="token punctuation">(</span><span class="token string">"./Cargo.toml"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">unwrap</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> content<span class="token punctuation">)</span><span class="token punctuation">}</span></code></pre><p>Even though we know that the file would be present, the compiler has no way of knowing that. So we use <code>unwrap</code> to tell the compiler to trust us and return the value inside. If the <code>read_to_string</code> function returns an <code>Ok()</code> value, <code>unwrap</code> will get the contents of <code>Ok</code> and assign it to the <code>content</code> variable. If it returns an error, it will “panic”. Panic either terminates the program or exits the current thread.</p><p>Note that <code>unwrap</code> is used in quite a lot of Rust examples to skip error handling. This is mostly done for convenience and shouldn’t be used in real code as it is.</p><h2 id="Terminate-the-program"><a href="#Terminate-the-program" class="headerlink" title="Terminate the program"></a>Terminate the program</h2><p>Some errors cannot be handled or recovered from. In these cases, it’s better to <em>fail fast</em> by terminating the program.</p><p>Let’s use the same example as above - we’re reading a file which we’re sure to be present. Let’s imagine that, for this program, that file is absolutely important without which it won’t work properly. If for some reason, this file is absent, it’s better to terminate the program.</p><p>We can use <code>unwrap</code> as before or use <code>expect</code> - it’s same as <code>unwrap</code> but lets us add extra error message.</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">use</span> std<span class="token punctuation">:</span><span class="token punctuation">:</span>fs<span class="token punctuation">;</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> content <span class="token operator">=</span> fs<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">read_to_string</span><span class="token punctuation">(</span><span class="token string">"./Cargo.toml"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">expect</span><span class="token punctuation">(</span><span class="token string">"Can't read Cargo.toml"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> content<span class="token punctuation">)</span><span class="token punctuation">}</span></code></pre><p>See also: <a href="https://doc.rust-lang.org/std/macro.panic.html" target="_blank" rel="noopener"><code>panic!</code></a></p><h2 id="Use-a-fallback-value"><a href="#Use-a-fallback-value" class="headerlink" title="Use a fallback value"></a>Use a fallback value</h2><p>In some cases, you can handle the error by falling back to a default value.</p><p>For example, let’s say we’re writing a server and the port it listens to can be configured using an environment variable. If the environment variable is not set, accessing that value would result in an error. But we can easily handle that by falling back to a default value.</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">use</span> std<span class="token punctuation">:</span><span class="token punctuation">:</span>env<span class="token punctuation">;</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> port <span class="token operator">=</span> env<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">var</span><span class="token punctuation">(</span><span class="token string">"PORT"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">unwrap_or</span><span class="token punctuation">(</span><span class="token string">"3000"</span><span class="token punctuation">.</span><span class="token function">to_string</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> port<span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>Here, we’ve used a variation of <code>unwrap</code> called <code>unwrap_or</code> which lets us supply default values.</p><p>See also: <a href="https://doc.rust-lang.org/std/result/enum.Result.html#method.unwrap_or_else" target="_blank" rel="noopener"><code>unwrap_or_else</code></a>, <a href="https://doc.rust-lang.org/std/result/enum.Result.html#method.unwrap_or_default" target="_blank" rel="noopener"><code>unwrap_or_default</code></a></p><h2 id="Bubble-up-the-error"><a href="#Bubble-up-the-error" class="headerlink" title="Bubble up the error"></a>Bubble up the error</h2><p>When you don’t have enough context to handle the error, you can bubble up (propagate) the error to the caller function.</p><p>Here’s a contrived example which uses a webservice to get the current year:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">use</span> std<span class="token punctuation">:</span><span class="token punctuation">:</span>collections<span class="token punctuation">:</span><span class="token punctuation">:</span>HashMap<span class="token punctuation">;</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">match</span> <span class="token function">get_current_date</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token function">Ok</span><span class="token punctuation">(</span>date<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"We've time travelled to {}!!"</span><span class="token punctuation">,</span> date<span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token function">Err</span><span class="token punctuation">(</span>e<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token function">eprintln!</span><span class="token punctuation">(</span><span class="token string">"Oh noes, we don't know which era we're in! :( \n  {}"</span><span class="token punctuation">,</span> e<span class="token punctuation">)</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token keyword">fn</span> <span class="token function">get_current_date</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">-></span> Result<span class="token operator">&lt;</span>String<span class="token punctuation">,</span> reqwest<span class="token punctuation">:</span><span class="token punctuation">:</span>Error<span class="token operator">></span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> url <span class="token operator">=</span> <span class="token string">"https://postman-echo.com/time/object"</span><span class="token punctuation">;</span>  <span class="token keyword">let</span> result <span class="token operator">=</span> reqwest<span class="token punctuation">:</span><span class="token punctuation">:</span>blocking<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">get</span><span class="token punctuation">(</span>url<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">let</span> response <span class="token operator">=</span> <span class="token keyword">match</span> result <span class="token punctuation">{</span>    <span class="token function">Ok</span><span class="token punctuation">(</span>res<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> res<span class="token punctuation">,</span>    <span class="token function">Err</span><span class="token punctuation">(</span>err<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token keyword">return</span> <span class="token function">Err</span><span class="token punctuation">(</span>err<span class="token punctuation">)</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">;</span>  <span class="token keyword">let</span> body <span class="token operator">=</span> response<span class="token punctuation">.</span>json<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token operator">&lt;</span>HashMap<span class="token operator">&lt;</span>String<span class="token punctuation">,</span> i32<span class="token operator">>></span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">let</span> json <span class="token operator">=</span> <span class="token keyword">match</span> body <span class="token punctuation">{</span>    <span class="token function">Ok</span><span class="token punctuation">(</span>json<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> json<span class="token punctuation">,</span>    <span class="token function">Err</span><span class="token punctuation">(</span>err<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token keyword">return</span> <span class="token function">Err</span><span class="token punctuation">(</span>err<span class="token punctuation">)</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">;</span>  <span class="token keyword">let</span> date <span class="token operator">=</span> json<span class="token punctuation">[</span><span class="token string">"years"</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">to_string</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">Ok</span><span class="token punctuation">(</span>date<span class="token punctuation">)</span><span class="token punctuation">}</span></code></pre><p>There are two function calls inside the <code>get_current_date</code> function (<code>get</code> and <code>json</code>) that return <code>Result</code> values. Since <code>get_current_date</code> doesn’t have context of what to do when they return errors, it uses pattern matching to propagate the errors to <code>main</code>.</p><p>Using pattern matching to handle multiple or nested errors can make your code “noisy”. Instead, we can rewrite the above code using the <a href="https://doc.rust-lang.org/book/ch09-02-recoverable-errors-with-result.html#a-shortcut-for-propagating-errors-the--operator" target="_blank" rel="noopener"><code>?</code> operator</a>:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">use</span> std<span class="token punctuation">:</span><span class="token punctuation">:</span>collections<span class="token punctuation">:</span><span class="token punctuation">:</span>HashMap<span class="token punctuation">;</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">match</span> <span class="token function">get_current_date</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token function">Ok</span><span class="token punctuation">(</span>date<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"We've time travelled to {}!!"</span><span class="token punctuation">,</span> date<span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token function">Err</span><span class="token punctuation">(</span>e<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token function">eprintln!</span><span class="token punctuation">(</span><span class="token string">"Oh noes, we don't know which era we're in! :( \n  {}"</span><span class="token punctuation">,</span> e<span class="token punctuation">)</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token keyword">fn</span> <span class="token function">get_current_date</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">-></span> Result<span class="token operator">&lt;</span>String<span class="token punctuation">,</span> reqwest<span class="token punctuation">:</span><span class="token punctuation">:</span>Error<span class="token operator">></span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> url <span class="token operator">=</span> <span class="token string">"https://postman-echo.com/time/object"</span><span class="token punctuation">;</span>  <span class="token keyword">let</span> res <span class="token operator">=</span> reqwest<span class="token punctuation">:</span><span class="token punctuation">:</span>blocking<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">get</span><span class="token punctuation">(</span>url<span class="token punctuation">)</span>?<span class="token punctuation">.</span>json<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token operator">&lt;</span>HashMap<span class="token operator">&lt;</span>String<span class="token punctuation">,</span> i32<span class="token operator">>></span><span class="token punctuation">(</span><span class="token punctuation">)</span>?<span class="token punctuation">;</span>  <span class="token keyword">let</span> date <span class="token operator">=</span> res<span class="token punctuation">[</span><span class="token string">"years"</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">to_string</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">Ok</span><span class="token punctuation">(</span>date<span class="token punctuation">)</span><span class="token punctuation">}</span></code></pre><p>This looks much cleaner!</p><p>The <code>?</code> operator is similar to <code>unwrap</code> but instead of panicking, it propagates the error to the calling function. One thing to keep in mind is that we can use the <code>?</code> operator only for functions that return a <code>Result</code> or <code>Option</code> type.</p><h2 id="Bubble-up-multiple-errors"><a href="#Bubble-up-multiple-errors" class="headerlink" title="Bubble up multiple errors"></a>Bubble up multiple errors</h2><p>In the previous example, the <code>get</code> and <code>json</code> functions return a <code>reqwest::Error</code> error which we’ve propagated using the <code>?</code> operator. But what if we’ve another function call that returned a different error value?</p><p>Let’s extend the previous example by returning a formatted date instead of the year:</p><pre class=" language-diff"><code class="language-diff"><span class="token inserted">+ use chrono::NaiveDate;</span>  use std::collections::HashMap;  fn main() {    match get_current_date() {      Ok(date) => println!("We've time travelled to {}!!", date),      Err(e) => eprintln!("Oh noes, we don't know which era we're in! :( \n  {}", e),    }  }  fn get_current_date() -> Result&lt;String, reqwest::Error> {    let url = "https://postman-echo.com/time/object";    let res = reqwest::blocking::get(url)?.json::&lt;HashMap&lt;String, i32>>()?;<span class="token deleted">-   let date = res["years"].to_string();</span><span class="token inserted">+   let formatted_date = format!("{}-{}-{}", res["years"], res["months"] + 1, res["date"]);</span><span class="token inserted">+   let parsed_date = NaiveDate::parse_from_str(formatted_date.as_str(), "%Y-%m-%d")?;</span><span class="token inserted">+   let date = parsed_date.format("%Y %B %d").to_string();</span>    Ok(date)  }</code></pre><p>The above code won’t compile as <code>parse_from_str</code> returns a <code>chrono::format::ParseError</code> error and not <code>reqwest::Error</code>.</p><p>We can fix this by <code>Box</code>ing the errors:</p><pre class=" language-diff"><code class="language-diff">  use chrono::NaiveDate;  use std::collections::HashMap;  fn main() {    match get_current_date() {      Ok(date) => println!("We've time travelled to {}!!", date),      Err(e) => eprintln!("Oh noes, we don't know which era we're in! :( \n  {}", e),    }  }<span class="token deleted">- fn get_current_date() -> Result&lt;String, reqwest::Error> {</span><span class="token inserted">+ fn get_current_date() -> Result&lt;String, Box&lt;dyn std::error::Error>> {</span>    let url = "https://postman-echo.com/time/object";    let res = reqwest::blocking::get(url)?.json::&lt;HashMap&lt;String, i32>>()?;    let formatted_date = format!("{}-{}-{}", res["years"], res["months"] + 1, res["date"]);    let parsed_date = NaiveDate::parse_from_str(formatted_date.as_str(), "%Y-%m-%d")?;    let date = parsed_date.format("%Y %B %d").to_string();    Ok(date)  }</code></pre><p>Returning a trait object <code>Box&lt;dyn std::error::Error&gt;</code> is very convenient when we want to return multiple errors!</p><p>See also: <a href="https://github.com/dtolnay/anyhow" target="_blank" rel="noopener"><code>anyhow</code></a>, <a href="https://github.com/yaahc/eyre" target="_blank" rel="noopener"><code>eyre</code></a></p><h2 id="Match-boxed-errors"><a href="#Match-boxed-errors" class="headerlink" title="Match boxed errors"></a>Match boxed errors</h2><p>So far, we’ve only printed the errors in the <code>main</code> function but not handled them. If we want to handle and recover from boxed errors, we need to “downcast” them:</p><pre class=" language-diff"><code class="language-diff">  use chrono::NaiveDate;  use std::collections::HashMap;  fn main() {    match get_current_date() {      Ok(date) => println!("We've time travelled to {}!!", date),<span class="token deleted">-     Err(e) => eprintln!("Oh noes, we don't know which era we're in! :( \n  {}", e),</span><span class="token inserted">+     Err(e) => {</span><span class="token inserted">+       eprintln!("Oh noes, we don't know which era we're in! :(");</span><span class="token inserted">+       if let Some(err) = e.downcast_ref::&lt;reqwest::Error>() {</span><span class="token inserted">+         eprintln!("Request Error: {}", err)</span><span class="token inserted">+       } else if let Some(err) = e.downcast_ref::&lt;chrono::format::ParseError>() {</span><span class="token inserted">+         eprintln!("Parse Error: {}", err)</span><span class="token inserted">+       }</span><span class="token inserted">+     }</span>    }  }  fn get_current_date() -> Result&lt;String, Box&lt;dyn std::error::Error>> {    let url = "https://postman-echo.com/time/object";    let res = reqwest::blocking::get(url)?.json::&lt;HashMap&lt;String, i32>>()?;    let formatted_date = format!("{}-{}-{}", res["years"], res["months"] + 1, res["date"]);    let parsed_date = NaiveDate::parse_from_str(formatted_date.as_str(), "%Y-%m-%d")?;    let date = parsed_date.format("%Y %B %d").to_string();    Ok(date)  }</code></pre><p>Notice how we need to be aware of the implementation details (different errors inside) of <code>get_current_date</code> to be able to downcast them inside <code>main</code>.</p><p>See also: <a href="https://doc.rust-lang.org/std/error/trait.Error.html#method.downcast" target="_blank" rel="noopener"><code>downcast</code></a>, <a href="https://doc.rust-lang.org/std/error/trait.Error.html#method.downcast_mut" target="_blank" rel="noopener"><code>downcast_mut</code></a></p><h2 id="Applications-vs-Libraries"><a href="#Applications-vs-Libraries" class="headerlink" title="Applications vs Libraries"></a>Applications vs Libraries</h2><p>As mentioned previously, the downside to boxed errors is that if we want to handle the underlying errors, we need to be aware of the implementation details. When we return something as <code>Box&lt;dyn std::error::Error&gt;</code>, the concrete type information is erased. To handle the different errors in different ways, we need to downcast them to concrete types and this casting can fail at runtime.</p><p>However, saying something is a “downside” is not very useful without context. A good rule of thumb is to question whether the code you’re writing is an “application” or a “library”:</p><h3 id="Application"><a href="#Application" class="headerlink" title="Application"></a>Application</h3><ul><li>The code you’re writing would be used by end users.</li><li>Most errors generated by application code won’t be handled but instead logged or reported to the user.</li><li>It’s okay to use boxed errors.</li></ul><h3 id="Library"><a href="#Library" class="headerlink" title="Library"></a>Library</h3><ul><li>The code you’re writing would be consumed by other code. A “library” can be open source crates, internal libraries etc</li><li>Errors are part of your library’s API, so your consumers know what errors to expect and recover from.</li><li>Errors from your library are often handled by your consumers so they need to be structured and easy to perform <a href="https://doc.rust-lang.org/1.30.0/book/2018-edition/ch06-02-match.html#matches-are-exhaustive" target="_blank" rel="noopener">exhaustive match</a> on.</li><li>If you return boxed errors, then your consumers need to be aware of the errors created by your code, your dependencies, and so on!</li><li>Instead of boxed errors, we can return custom errors.</li></ul><h2 id="Create-custom-errors"><a href="#Create-custom-errors" class="headerlink" title="Create custom errors"></a>Create custom errors</h2><p>For library code, we can convert all the errors to our own custom error and propagate them instead of boxed errors. In our example, we currently have two errors - <code>reqwest::Error</code> and <code>chrono::format::ParseError</code>. We can convert them to <code>MyCustomError::HttpError</code> and <code>MyCustomError::ParseError</code> respectively.</p><p>Let’s start by creating an enum to hold our two error variants:</p><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// error.rs</span><span class="token keyword">pub</span> <span class="token keyword">enum</span> MyCustomError <span class="token punctuation">{</span>  HttpError<span class="token punctuation">,</span>  ParseError<span class="token punctuation">,</span><span class="token punctuation">}</span></code></pre><p>The <a href="https://doc.rust-lang.org/std/error/trait.Error.html" target="_blank" rel="noopener"><code>Error</code></a> trait requires us to implement the <a href="https://doc.rust-lang.org/std/fmt/trait.Debug.html" target="_blank" rel="noopener"><code>Debug</code></a> and <a href="https://doc.rust-lang.org/std/fmt/trait.Display.html" target="_blank" rel="noopener"><code>Display</code></a> traits:</p><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// error.rs</span><span class="token keyword">use</span> std<span class="token punctuation">:</span><span class="token punctuation">:</span>fmt<span class="token punctuation">;</span><span class="token attribute attr-name">#[derive(Debug)]</span><span class="token keyword">pub</span> <span class="token keyword">enum</span> MyCustomError <span class="token punctuation">{</span>  HttpError<span class="token punctuation">,</span>  ParseError<span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token keyword">impl</span> std<span class="token punctuation">:</span><span class="token punctuation">:</span>error<span class="token punctuation">:</span><span class="token punctuation">:</span>Error <span class="token keyword">for</span> MyCustomError <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token keyword">impl</span> fmt<span class="token punctuation">:</span><span class="token punctuation">:</span>Display <span class="token keyword">for</span> MyCustomError <span class="token punctuation">{</span>  <span class="token keyword">fn</span> <span class="token function">fmt</span><span class="token punctuation">(</span><span class="token operator">&amp;</span><span class="token keyword">self</span><span class="token punctuation">,</span> f<span class="token punctuation">:</span> <span class="token operator">&amp;</span><span class="token keyword">mut</span> fmt<span class="token punctuation">:</span><span class="token punctuation">:</span>Formatter<span class="token punctuation">)</span> <span class="token punctuation">-></span> fmt<span class="token punctuation">:</span><span class="token punctuation">:</span>Result <span class="token punctuation">{</span>    <span class="token keyword">match</span> <span class="token keyword">self</span> <span class="token punctuation">{</span>      MyCustomError<span class="token punctuation">:</span><span class="token punctuation">:</span>HttpError <span class="token operator">=</span><span class="token operator">></span> <span class="token function">write!</span><span class="token punctuation">(</span>f<span class="token punctuation">,</span> <span class="token string">"HTTP Error"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>      MyCustomError<span class="token punctuation">:</span><span class="token punctuation">:</span>ParseError <span class="token operator">=</span><span class="token operator">></span> <span class="token function">write!</span><span class="token punctuation">(</span>f<span class="token punctuation">,</span> <span class="token string">"Parse Error"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token punctuation">}</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><p>We’ve created our own custom error!</p><p>This is obviously a simple example as the error variants don’t contain much information about the error. But this should be sufficient as a starting point for creating more complex and realistic custom errors. Here are some real life examples: <a href="https://github.com/BurntSushi/ripgrep/blob/12.1.1/crates/regex/src/error.rs" target="_blank" rel="noopener">ripgrep</a>, <a href="https://github.com/seanmonstar/reqwest/blob/v0.10.7/src/error.rs" target="_blank" rel="noopener">reqwest</a>, <a href="https://github.com/BurntSushi/rust-csv/blob/master/src/error.rs" target="_blank" rel="noopener">csv</a> and <a href="https://github.com/serde-rs/json/blob/master/src/error.rs" target="_blank" rel="noopener">serde_json</a></p><p>See also: <a href="https://github.com/dtolnay/thiserror" target="_blank" rel="noopener"><code>thiserror</code></a>, <a href="https://github.com/shepmaster/snafu" target="_blank" rel="noopener"><code>snafu</code></a></p><h2 id="Bubble-up-custom-errors"><a href="#Bubble-up-custom-errors" class="headerlink" title="Bubble up custom errors"></a>Bubble up custom errors</h2><p>Let’s update our code to return the custom errors we just created:</p><pre class=" language-diff"><code class="language-diff">  // main.rs<span class="token inserted">+ mod error;</span>  use chrono::NaiveDate;<span class="token inserted">+ use error::MyCustomError;</span>  use std::collections::HashMap;  fn main() {    // skipped, will get back later  }<span class="token deleted">- fn get_current_date() -> Result&lt;String, Box&lt;dyn std::error::Error>> {</span><span class="token inserted">+ fn get_current_date() -> Result&lt;String, MyCustomError> {</span>    let url = "https://postman-echo.com/time/object";<span class="token deleted">-   let res = reqwest::blocking::get(url)?.json::&lt;HashMap&lt;String, i32>>()?;</span><span class="token inserted">+   let res = reqwest::blocking::get(url)</span><span class="token inserted">+     .map_err(|_| MyCustomError::HttpError)?</span><span class="token inserted">+     .json::&lt;HashMap&lt;String, i32>>()</span><span class="token inserted">+     .map_err(|_| MyCustomError::HttpError)?;</span>    let formatted_date = format!("{}-{}-{}", res["years"], res["months"] + 1, res["date"]);<span class="token deleted">-   let parsed_date = NaiveDate::parse_from_str(formatted_date.as_str(), "%Y-%m-%d")?;</span><span class="token inserted">+   let parsed_date = NaiveDate::parse_from_str(formatted_date.as_str(), "%Y-%m-%d")</span><span class="token inserted">+     .map_err(|_| MyCustomError::ParseError)?;</span>    let date = parsed_date.format("%Y %B %d").to_string();    Ok(date)  }</code></pre><p>Notice how we’re using <code>map_err</code> to convert the error from one type to another type.</p><p>But things got verbose as a result - our function is littered with these <code>map_err</code> calls. We can implement the <a href="https://doc.rust-lang.org/std/convert/trait.From.html" target="_blank" rel="noopener"><code>From</code></a> trait to automatically coerce the error types when we use the <code>?</code> operator:</p><pre class=" language-diff"><code class="language-diff">  // error.rs  use std::fmt;  #[derive(Debug)]  pub enum MyCustomError {    HttpError,    ParseError,  }  impl std::error::Error for MyCustomError {}  impl fmt::Display for MyCustomError {    fn fmt(&amp;self, f: &amp;mut fmt::Formatter) -> fmt::Result {      match self {        MyCustomError::HttpError => write!(f, "HTTP Error"),        MyCustomError::ParseError => write!(f, "Parse Error"),      }    }  }<span class="token inserted">+ impl From&lt;reqwest::Error> for MyCustomError {</span><span class="token inserted">+   fn from(_: reqwest::Error) -> Self {</span><span class="token inserted">+     MyCustomError::HttpError</span><span class="token inserted">+   }</span><span class="token inserted">+ }</span><span class="token inserted">+ impl From&lt;chrono::format::ParseError> for MyCustomError {</span><span class="token inserted">+   fn from(_: chrono::format::ParseError) -> Self {</span><span class="token inserted">+     MyCustomError::ParseError</span><span class="token inserted">+   }</span><span class="token inserted">+ }</span></code></pre><pre class=" language-diff"><code class="language-diff">  // main.rs  mod error;  use chrono::NaiveDate;  use error::MyCustomError;  use std::collections::HashMap;  fn main() {    // skipped, will get back later  }  fn get_current_date() -> Result&lt;String, MyCustomError> {    let url = "https://postman-echo.com/time/object";<span class="token deleted">-   let res = reqwest::blocking::get(url)</span><span class="token deleted">-     .map_err(|_| MyCustomError::HttpError)?</span><span class="token deleted">-     .json::&lt;HashMap&lt;String, i32>>()</span><span class="token deleted">-     .map_err(|_| MyCustomError::HttpError)?;</span><span class="token inserted">+   let res = reqwest::blocking::get(url)?.json::&lt;HashMap&lt;String, i32>>()?;</span>    let formatted_date = format!("{}-{}-{}", res["years"], res["months"] + 1, res["date"]);<span class="token deleted">-   let parsed_date = NaiveDate::parse_from_str(formatted_date.as_str(), "%Y-%m-%d")</span><span class="token deleted">-     .map_err(|_| MyCustomError::ParseError)?;</span><span class="token inserted">+   let parsed_date = NaiveDate::parse_from_str(formatted_date.as_str(), "%Y-%m-%d")?;</span>    let date = parsed_date.format("%Y %B %d").to_string();    Ok(date)  }</code></pre><p>We’ve removed <code>map_err</code> and the code looks much cleaner!</p><p>However, <code>From</code> trait is not magic and there are times when we need to use <code>map_err</code>. In the above example, we’ve moved the type conversion from inside the <code>get_current_date</code> function to the <code>From&lt;X&gt; for MyCustomError</code> implementation. This works well when the information needed to convert from one error to <code>MyCustomError</code> can be obtained from the original error object. If not, we need to use <code>map_err</code> inside <code>get_current_date</code>.</p><h2 id="Match-custom-errors"><a href="#Match-custom-errors" class="headerlink" title="Match custom errors"></a>Match custom errors</h2><p>We’ve ignored the changes in <code>main</code> until now, here’s how we can handle the custom errors:</p><pre class=" language-diff"><code class="language-diff">  // main.rs  mod error;  use chrono::NaiveDate;  use error::MyCustomError;  use std::collections::HashMap;  fn main() {    match get_current_date() {      Ok(date) => println!("We've time travelled to {}!!", date),      Err(e) => {        eprintln!("Oh noes, we don't know which era we're in! :(");<span class="token deleted">-       if let Some(err) = e.downcast_ref::&lt;reqwest::Error>() {</span><span class="token deleted">-         eprintln!("Request Error: {}", err)</span><span class="token deleted">-       } else if let Some(err) = e.downcast_ref::&lt;chrono::format::ParseError>() {</span><span class="token deleted">-         eprintln!("Parse Error: {}", err)</span><span class="token deleted">-       }</span><span class="token inserted">+       match e {</span><span class="token inserted">+         MyCustomError::HttpError => eprintln!("Request Error: {}", e),</span><span class="token inserted">+         MyCustomError::ParseError => eprintln!("Parse Error: {}", e),</span><span class="token inserted">+       }</span>      }    }  }  fn get_current_date() -> Result&lt;String, MyCustomError> {    let url = "https://postman-echo.com/time/object";    let res = reqwest::blocking::get(url)?.json::&lt;HashMap&lt;String, i32>>()?;    let formatted_date = format!("{}-{}-{}", res["years"], res["months"] + 1, res["date"]);    let parsed_date = NaiveDate::parse_from_str(formatted_date.as_str(), "%Y-%m-%d")?;    let date = parsed_date.format("%Y %B %d").to_string();    Ok(date)  }</code></pre><p>Notice how unlike boxed errors, we can actually match on the variants inside <code>MyCustomError</code> enum.</p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>Thanks for reading! I hope this post was helpful in introducing the basics of error handling in Rust. I’ve added the examples to a <a href="https://github.com/sheshbabu/rust-error-handling-examples" target="_blank" rel="noopener">repo in GitHub</a> which you can use for practice. If you’ve more questions, please contact me at sheshbabu [at] gmail.com. Feel free to follow me in <a href="https://twitter.com/sheshbabu" target="_blank" rel="noopener">Twitter</a> for more posts like this :)</p>]]></content>
    
    <summary type="html">
    
      Easy to follow guide using practical examples
    
    </summary>
    
    
      <category term="Rust" scheme="https://www.sheshbabu.com/tags/Rust/"/>
    
      <category term="Rust Beginners" scheme="https://www.sheshbabu.com/tags/Rust-Beginners/"/>
    
  </entry>
  
  <entry>
    <title>Clear explanation of Rust’s module system</title>
    <link href="https://www.sheshbabu.com/posts/rust-module-system/"/>
    <id>https://www.sheshbabu.com/posts/rust-module-system/</id>
    <published>2020-07-19T02:38:14.000Z</published>
    <updated>2024-05-19T06:17:28.272Z</updated>
    
    <content type="html"><![CDATA[<p>Rust’s module system is surprisingly confusing and causes a lot of frustration for beginners.</p><p>In this post, I’ll explain the module system using practical examples so you get a clear understanding of how it works and can immediately start applying this in your projects.</p><p>Since Rust’s module system is quite unique, I request the reader to read this post with an open mind and resist comparing it with how modules work in other languages.</p><p>Let’s use this file structure to simulate a real world project:</p><pre class=" language-shell"><code class="language-shell">my_project├── Cargo.toml└─┬ src  ├── main.rs  ├── config.rs  ├─┬ routes  │ ├── health_route.rs  │ └── user_route.rs  └─┬ models    └── user_model.rs</code></pre><p>These are the different ways we should be able to consume our modules:</p><p><img src="/images/2020-rust-module-system/rust-module-system-1.png" alt=""></p><p>These 3 examples should be sufficient to explain how Rust’s module system works.</p><h2 id="Example-1"><a href="#Example-1" class="headerlink" title="Example 1"></a>Example 1</h2><p>Let’s start with the first example - importing <code>config.rs</code> in <code>main.rs</code>.</p><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// main.rs</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"main"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// config.rs</span><span class="token keyword">fn</span> <span class="token function">print_config</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"config"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>The first mistake that everyone makes is just because we have files like <code>config.rs</code>, <code>health_route.rs</code> etc, we think that these files are <code>modules</code> and we can import them from other files.</p><p>Here’s what we see (file system tree) and what the compiler sees (module tree):</p><p><img src="/images/2020-rust-module-system/rust-module-system-2.png" alt=""></p><p>Surprisingly, the compiler only sees the <code>crate</code> module which is our <code>main.rs</code> file. This is because we need to explicitly build the module tree in Rust - there’s no implicit mapping between file system tree to module tree.</p><blockquote><p>We need to explicitly build the module tree in Rust, there’s no implicit mapping to file system</p></blockquote><p>To add a file to the module tree, we need to declare that file as a submodule using the <code>mod</code> keyword. The next thing that confuses people is that you would assume we declare a file as module in the same file. But we need to declare this in a different file! Since we only have <code>main.rs</code> in the module tree, let’s declare <code>config.rs</code> as a submodule in <code>main.rs</code>.</p><blockquote><p>The mod keyword declares a submodule</p></blockquote><p>The <code>mod</code> keyword has this syntax:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">mod</span> my_module<span class="token punctuation">;</span></code></pre><p>Here, the compiler looks for <code>my_module.rs</code> or <code>my_module/mod.rs</code> in the same directory.</p><pre class=" language-shell"><code class="language-shell">my_project├── Cargo.toml└─┬ src  ├── main.rs  └── my_module.rsormy_project├── Cargo.toml└─┬ src  ├── main.rs  └─┬ my_module    └── mod.rs</code></pre><p>Since <code>main.rs</code> and <code>config.rs</code> are in the same directory, let’s declare the config module as follows:</p><pre class=" language-diff"><code class="language-diff">// main.rs<span class="token inserted">+ mod config;</span>fn main() {<span class="token inserted">+ config::print_config();</span>  println!("main");}</code></pre><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// config.rs</span><span class="token keyword">fn</span> <span class="token function">print_config</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"config"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>We’re accessing the <code>print_config</code> function using the <code>::</code> syntax.</p><p>Here’s how the module tree looks like:</p><p><img src="/images/2020-rust-module-system/rust-module-system-3.png" alt=""></p><p>We’ve successfully declared the <code>config</code> module! But this is not sufficient to be able to call the <code>print_config</code> function inside <code>config.rs</code>. Almost everything in Rust is private by default, we need to make the function public using the <code>pub</code> keyword:</p><blockquote><p>The pub keyword makes things public</p></blockquote><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// main.rs</span><span class="token keyword">mod</span> config<span class="token punctuation">;</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  config<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">print_config</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"main"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><pre class=" language-diff"><code class="language-diff">// config.rs<span class="token deleted">- fn print_config() {</span><span class="token inserted">+ pub fn print_config() {</span>  println!("config");}</code></pre><p>Now, this works. We’ve successfully called a function defined in a different file!</p><h2 id="Example-2"><a href="#Example-2" class="headerlink" title="Example 2"></a>Example 2</h2><p>Let’s try calling the <code>print_health_route</code> function defined in <code>routes/health_route.rs</code> from <code>main.rs</code>.</p><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// main.rs</span><span class="token keyword">mod</span> config<span class="token punctuation">;</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  config<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">print_config</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"main"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// routes/health_route.rs</span><span class="token keyword">fn</span> <span class="token function">print_health_route</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"health_route"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>As we discussed earlier, we can use the <code>mod</code> keyword only for <code>my_module.rs</code> or <code>my_module/mod.rs</code> in the same directory.</p><p>So in order to call functions inside <code>routes/health_route.rs</code> from <code>main.rs</code>, we need to do the following things:</p><ul><li>Create a file named <code>routes/mod.rs</code> and declare the <code>routes</code> submodule in <code>main.rs</code></li><li>Declare the <code>health_route</code> submodule in <code>routes/mod.rs</code> and make it public</li><li>Make the functions inside <code>health_route.rs</code> public</li></ul><pre class=" language-diff"><code class="language-diff">my_project├── Cargo.toml└─┬ src  ├── main.rs  ├── config.rs  ├─┬ routes<span class="token inserted">+ │ ├── mod.rs</span>  │ ├── health_route.rs  │ └── user_route.rs  └─┬ models    └── user_model.rs</code></pre><pre class=" language-diff"><code class="language-diff">// main.rsmod config;<span class="token inserted">+ mod routes;</span>fn main() {<span class="token inserted">+ routes::health_route::print_health_route();</span>  config::print_config();  println!("main");}</code></pre><pre class=" language-diff"><code class="language-diff">// routes/mod.rs<span class="token inserted">+ pub mod health_route;</span></code></pre><pre class=" language-diff"><code class="language-diff">// routes/health_route.rs<span class="token deleted">- fn print_health_route() {</span><span class="token inserted">+ pub fn print_health_route() {</span>  println!("health_route");}</code></pre><p>Here’s how the module tree looks like:</p><p><img src="/images/2020-rust-module-system/rust-module-system-4.png" alt=""></p><p>We can now call a function defined in a file inside a folder.</p><h2 id="Example-3"><a href="#Example-3" class="headerlink" title="Example 3"></a>Example 3</h2><p>Let’s try calling from <code>main.rs =&gt; routes/user_route.rs =&gt; models/user_model.rs</code></p><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// main.rs</span><span class="token keyword">mod</span> config<span class="token punctuation">;</span><span class="token keyword">mod</span> routes<span class="token punctuation">;</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  routes<span class="token punctuation">:</span><span class="token punctuation">:</span>health_route<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">print_health_route</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  config<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">print_config</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"main"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// routes/user_route.rs</span><span class="token keyword">fn</span> <span class="token function">print_user_route</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"user_route"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// models/user_model.rs</span><span class="token keyword">fn</span> <span class="token function">print_user_model</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"user_model"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>We want to call the function <code>print_user_model</code> from <code>print_user_route</code> from <code>main</code>.</p><p>Let’s make the same changes as before - declaring submodules, making functions public and adding the <code>mod.rs</code> file.</p><pre class=" language-diff"><code class="language-diff">my_project├── Cargo.toml└─┬ src  ├── main.rs  ├── config.rs  ├─┬ routes  │ ├── mod.rs  │ ├── health_route.rs  │ └── user_route.rs  └─┬ models<span class="token inserted">+   ├── mod.rs</span>    └── user_model.rs</code></pre><pre class=" language-diff"><code class="language-diff">// main.rsmod config;mod routes;<span class="token inserted">+ mod models;</span>fn main() {  routes::health_route::print_health_route();<span class="token inserted">+ routes::user_route::print_user_route();</span>  config::print_config();  println!("main");}</code></pre><pre class=" language-diff"><code class="language-diff">// routes/mod.rspub mod health_route;<span class="token inserted">+ pub mod user_route;</span></code></pre><pre class=" language-diff"><code class="language-diff">// routes/user_route.rs<span class="token deleted">- fn print_user_route() {</span><span class="token inserted">+ pub fn print_user_route() {</span>  println!("user_route");}</code></pre><pre class=" language-diff"><code class="language-diff">// models/mod.rs<span class="token inserted">+ pub mod user_model;</span></code></pre><pre class=" language-diff"><code class="language-diff">// models/user_model.rs<span class="token deleted">- fn print_user_model() {</span><span class="token inserted">+ pub fn print_user_model() {</span>  println!("user_model");}</code></pre><p>Here’s how the module tree looks like:</p><p><img src="/images/2020-rust-module-system/rust-module-system-5.png" alt=""></p><p>Wait, we haven’t actually called <code>print_user_model</code> from <code>print_user_route</code>! So far, we’ve only called the functions defined in other modules from <code>main.rs</code>, how do we do that from other files?</p><p>If we look at our module tree, the <code>print_user_model</code> function sits in the <code>crate::models::user_model</code> path. So in order to use a module in files that are not <code>main.rs</code>, we should think in terms of the path necessary to reach that module in the module tree.</p><pre class=" language-diff"><code class="language-diff">// routes/user_route.rspub fn print_user_route() {<span class="token inserted">+ crate::models::user_model::print_user_model();</span>  println!("user_route");}</code></pre><p>We’ve successfully called a function defined in a file from a file that’s not <code>main.rs</code>.</p><h2 id="super"><a href="#super" class="headerlink" title="super"></a>super</h2><p>The fully qualified name gets too lengthy if our file organization is multiple directories deep. Let’s say for whatever reason, we want to call <code>print_health_route</code> from <code>print_user_route</code>. These are under the paths <code>crate::routes::health_route</code> and <code>crate::routes::user_route</code> respectively.</p><p>We can call it by using the fully qualified name <code>crate::routes::health_route::print_health_route()</code> but we can also use a relative path <code>super::health_route::print_health_route();</code>. Notice that we’ve used <code>super</code> to refer to the parent scope.</p><blockquote><p>The super keyword in module path refers to the parent scope</p></blockquote><pre class=" language-rust"><code class="language-rust"><span class="token keyword">pub</span> <span class="token keyword">fn</span> <span class="token function">print_user_route</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">crate</span><span class="token punctuation">:</span><span class="token punctuation">:</span>routes<span class="token punctuation">:</span><span class="token punctuation">:</span>health_route<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">print_health_route</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token comment" spellcheck="true">// can also be called using</span>  <span class="token keyword">super</span><span class="token punctuation">:</span><span class="token punctuation">:</span>health_route<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">print_health_route</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"user_route"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><h2 id="use"><a href="#use" class="headerlink" title="use"></a>use</h2><p>It would be tedious to use the fully qualified name or even the relative name in the above examples. In order to shorten the names, we can use the <code>use</code> keyword to bind the path to a new name or alias.</p><blockquote><p>The use keyword is used to shorten the module path</p></blockquote><pre class=" language-rust"><code class="language-rust"><span class="token keyword">pub</span> <span class="token keyword">fn</span> <span class="token function">print_user_route</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">crate</span><span class="token punctuation">:</span><span class="token punctuation">:</span>models<span class="token punctuation">:</span><span class="token punctuation">:</span>user_model<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">print_user_model</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"user_route"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>The above code can be refactored as:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">use</span> <span class="token keyword">crate</span><span class="token punctuation">:</span><span class="token punctuation">:</span>models<span class="token punctuation">:</span><span class="token punctuation">:</span>user_model<span class="token punctuation">:</span><span class="token punctuation">:</span>print_user_model<span class="token punctuation">;</span><span class="token keyword">pub</span> <span class="token keyword">fn</span> <span class="token function">print_user_route</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token function">print_user_model</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"user_route"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>Instead of using the name <code>print_user_model</code>, we can also alias it to something else:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">use</span> <span class="token keyword">crate</span><span class="token punctuation">:</span><span class="token punctuation">:</span>models<span class="token punctuation">:</span><span class="token punctuation">:</span>user_model<span class="token punctuation">:</span><span class="token punctuation">:</span>print_user_model <span class="token keyword">as</span> log_user_model<span class="token punctuation">;</span><span class="token keyword">pub</span> <span class="token keyword">fn</span> <span class="token function">print_user_route</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token function">log_user_model</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"user_route"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><h2 id="External-modules"><a href="#External-modules" class="headerlink" title="External modules"></a>External modules</h2><p>Dependencies added to <code>Cargo.toml</code> are available globally to all modules inside the project. We don’t need to explicitly import or declare anything to use a dependency.</p><blockquote><p>External dependencies are globally available to all modules inside a project</p></blockquote><p>For example, let’s say we added the <a href="https://crates.io/crates/rand" target="_blank" rel="noopener">rand</a> crate to our project. We can use it in our code directly as:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">pub</span> <span class="token keyword">fn</span> <span class="token function">print_health_route</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> random_number<span class="token punctuation">:</span> u8 <span class="token operator">=</span> rand<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">random</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> random_number<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"health_route"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>We can also use <code>use</code> to shorten the path:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">use</span> rand<span class="token punctuation">:</span><span class="token punctuation">:</span>random<span class="token punctuation">;</span><span class="token keyword">pub</span> <span class="token keyword">fn</span> <span class="token function">print_health_route</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> random_number<span class="token punctuation">:</span> u8 <span class="token operator">=</span> <span class="token function">random</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> random_number<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"health_route"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><h2 id="Summary"><a href="#Summary" class="headerlink" title="Summary"></a>Summary</h2><ul><li>The module system is explicit - there’s no 1:1 mapping with file system</li><li>We declare a file as module in its parent, not in itself</li><li>The <code>mod</code> keyword is used to declare submodules</li><li>We need to explicitly declare functions, structs etc as public so they can be consumed in other modules</li><li>The <code>pub</code> keyword makes things public</li><li>The <code>use</code> keyword is used to shorten the module path</li><li>We don’t need to explicitly declare 3rd party modules</li></ul><p>Thanks for reading! Feel free to follow me in <a href="https://twitter.com/sheshbabu" target="_blank" rel="noopener">Twitter</a> for more posts like this :)</p>]]></content>
    
    <summary type="html">
    
      Easy to understand explanation using visual aids and practical examples
    
    </summary>
    
    
      <category term="Rust" scheme="https://www.sheshbabu.com/tags/Rust/"/>
    
      <category term="Rust Beginners" scheme="https://www.sheshbabu.com/tags/Rust-Beginners/"/>
    
  </entry>
  
  <entry>
    <title>Rust for JavaScript Developers - Pattern Matching and Enums</title>
    <link href="https://www.sheshbabu.com/posts/rust-for-javascript-developers-pattern-matching-and-enums/"/>
    <id>https://www.sheshbabu.com/posts/rust-for-javascript-developers-pattern-matching-and-enums/</id>
    <published>2020-07-12T05:48:03.000Z</published>
    <updated>2024-05-19T06:17:28.272Z</updated>
    
    <content type="html"><![CDATA[<p>This is the fourth part in a series about introducing the Rust language to JavaScript developers. Here are all the chapters:</p><ol><li><a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-tooling-ecosystem-overview/">Tooling Ecosystem Overview</a></li><li><a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-variables-and-data-types/">Variables and Data Types</a></li><li><a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-functions-and-control-flow/">Functions and Control Flow</a></li><li><a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-pattern-matching-and-enums/">Pattern Matching and Enums</a></li></ol><h2 id="Pattern-Matching"><a href="#Pattern-Matching" class="headerlink" title="Pattern Matching"></a>Pattern Matching</h2><p>To understand Pattern Matching, let’s start with something familiar in JavaScript - Switch Case.</p><p>Here’s a simple example that uses <code>switch case</code> in JavaScript:</p><pre class=" language-javascript"><code class="language-javascript"><span class="token keyword">function</span> <span class="token function">print_color</span><span class="token punctuation">(</span>color<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">switch</span> <span class="token punctuation">(</span>color<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">case</span> <span class="token string">"rose"</span><span class="token punctuation">:</span>      console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"roses are red,"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>      <span class="token keyword">break</span><span class="token punctuation">;</span>    <span class="token keyword">case</span> <span class="token string">"violet"</span><span class="token punctuation">:</span>      console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"violets are blue,"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>      <span class="token keyword">break</span><span class="token punctuation">;</span>    <span class="token keyword">default</span><span class="token punctuation">:</span>      console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"sugar is sweet, and so are you."</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token function">print_color</span><span class="token punctuation">(</span><span class="token string">"rose"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// roses are red,</span><span class="token function">print_color</span><span class="token punctuation">(</span><span class="token string">"violet"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// violets are blue,</span><span class="token function">print_color</span><span class="token punctuation">(</span><span class="token string">"you"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// sugar is sweet, and so are you.</span></code></pre><p>Here’s the equivalent Rust code:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">fn</span> <span class="token function">print_color</span><span class="token punctuation">(</span>color<span class="token punctuation">:</span> <span class="token operator">&amp;</span>str<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">match</span> color <span class="token punctuation">{</span>    <span class="token string">"rose"</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"roses are red,"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    <span class="token string">"violet"</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"violets are blue,"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    _ <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"sugar is sweet, and so are you."</span><span class="token punctuation">)</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token function">print_color</span><span class="token punctuation">(</span><span class="token string">"rose"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// roses are red,</span>  <span class="token function">print_color</span><span class="token punctuation">(</span><span class="token string">"violet"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// violets are blue,</span>  <span class="token function">print_color</span><span class="token punctuation">(</span><span class="token string">"you"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// sugar is sweet, and so are you.</span><span class="token punctuation">}</span></code></pre><p>Most of the code should be immediately understandable. The <code>match</code> expression has the following signature:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">match</span> VALUE <span class="token punctuation">{</span>  PATTERN1 <span class="token operator">=</span><span class="token operator">></span> EXPRESSION1<span class="token punctuation">,</span>  PATTERN2 <span class="token operator">=</span><span class="token operator">></span> EXPRESSION2<span class="token punctuation">,</span>  PATTERN3 <span class="token operator">=</span><span class="token operator">></span> EXPRESSION3<span class="token punctuation">,</span><span class="token punctuation">}</span></code></pre><p>The fat arrow <code>=&gt;</code> syntax might trip us up because of the similarities with JavaScript arrow functions but they’re unrelated. The last pattern that uses underscore <code>_</code> is called the catchall pattern and is similar to the default case in switch case. Each <code>PATTERN =&gt; EXPRESSION</code> combination is called a <code>match arm</code>.</p><p>The above example doesn’t really convey how useful pattern matching is - it just looks like switch case with a different syntax and a fancy name. Let’s talk about destructuring and enums to understand why pattern matching is useful.</p><h2 id="Destructuring"><a href="#Destructuring" class="headerlink" title="Destructuring"></a>Destructuring</h2><p>Destructuring is the process of extracting the inner fields of an array or struct into separate variables. If you have used destructuring in JavaScript, it is very similar in Rust.</p><p>Here’s an example in JavaScript:</p><pre class=" language-javascript"><code class="language-javascript"><span class="token keyword">let</span> rgb <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token number">96</span><span class="token punctuation">,</span> <span class="token number">172</span><span class="token punctuation">,</span> <span class="token number">57</span><span class="token punctuation">]</span><span class="token punctuation">;</span><span class="token keyword">let</span> <span class="token punctuation">[</span>red<span class="token punctuation">,</span> green<span class="token punctuation">,</span> blue<span class="token punctuation">]</span> <span class="token operator">=</span> rgb<span class="token punctuation">;</span>console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>red<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// 96</span>console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>green<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// 172</span>console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>blue<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// 57</span><span class="token keyword">let</span> person <span class="token operator">=</span> <span class="token punctuation">{</span> name<span class="token punctuation">:</span> <span class="token string">"shesh"</span><span class="token punctuation">,</span> city<span class="token punctuation">:</span> <span class="token string">"singapore"</span> <span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token keyword">let</span> <span class="token punctuation">{</span> name<span class="token punctuation">,</span> city <span class="token punctuation">}</span> <span class="token operator">=</span> person<span class="token punctuation">;</span>console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// name</span>console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>city<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// city</span></code></pre><p>Here’s the same example in Rust:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">struct</span> Person <span class="token punctuation">{</span>  name<span class="token punctuation">:</span> String<span class="token punctuation">,</span>  city<span class="token punctuation">:</span> String<span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> rgb <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token number">96</span><span class="token punctuation">,</span> <span class="token number">172</span><span class="token punctuation">,</span> <span class="token number">57</span><span class="token punctuation">]</span><span class="token punctuation">;</span>  <span class="token keyword">let</span> <span class="token punctuation">[</span>red<span class="token punctuation">,</span> green<span class="token punctuation">,</span> blue<span class="token punctuation">]</span> <span class="token operator">=</span> rgb<span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> red<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// 96</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> green<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// 172</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> blue<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// 57</span>  <span class="token keyword">let</span> person <span class="token operator">=</span> Person <span class="token punctuation">{</span>    name<span class="token punctuation">:</span> <span class="token string">"shesh"</span><span class="token punctuation">.</span><span class="token function">to_string</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    city<span class="token punctuation">:</span> <span class="token string">"singapore"</span><span class="token punctuation">.</span><span class="token function">to_string</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">;</span>  <span class="token keyword">let</span> Person <span class="token punctuation">{</span> name<span class="token punctuation">,</span> city <span class="token punctuation">}</span> <span class="token operator">=</span> person<span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> name<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// name</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> city<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// city</span><span class="token punctuation">}</span></code></pre><h2 id="Comparing-Structs"><a href="#Comparing-Structs" class="headerlink" title="Comparing Structs"></a>Comparing Structs</h2><p>It’s very common to write “if this then that” type of code. Combining destructuring and pattern matching allows us to write these type of logic in a very concise way.</p><p>Let’s take this following example in JavaScript. It’s contrived but you have probably written something like this sometime in your career:</p><pre class=" language-javascript"><code class="language-javascript"><span class="token keyword">const</span> point <span class="token operator">=</span> <span class="token punctuation">{</span> x<span class="token punctuation">:</span> <span class="token number">0</span><span class="token punctuation">,</span> y<span class="token punctuation">:</span> <span class="token number">30</span> <span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token keyword">const</span> <span class="token punctuation">{</span> x<span class="token punctuation">,</span> y <span class="token punctuation">}</span> <span class="token operator">=</span> point<span class="token punctuation">;</span><span class="token keyword">if</span> <span class="token punctuation">(</span>x <span class="token operator">===</span> <span class="token number">0</span> <span class="token operator">&amp;&amp;</span> y <span class="token operator">===</span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"both are zero"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>x <span class="token operator">===</span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token string">`x is zero and y is </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>y<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>y <span class="token operator">===</span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token string">`x is </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>x<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"> and y is zero`</span></span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>  console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token string">`x is </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>x<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"> and y is </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>y<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>Let’s write the same code in Rust using pattern matching:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">struct</span> Point <span class="token punctuation">{</span>  x<span class="token punctuation">:</span> i32<span class="token punctuation">,</span>  y<span class="token punctuation">:</span> i32<span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> point <span class="token operator">=</span> Point <span class="token punctuation">{</span> x<span class="token punctuation">:</span> <span class="token number">10</span><span class="token punctuation">,</span> y<span class="token punctuation">:</span> <span class="token number">0</span> <span class="token punctuation">}</span><span class="token punctuation">;</span>  <span class="token keyword">match</span> point <span class="token punctuation">{</span>    Point <span class="token punctuation">{</span> x<span class="token punctuation">:</span> <span class="token number">0</span><span class="token punctuation">,</span> y<span class="token punctuation">:</span> <span class="token number">0</span> <span class="token punctuation">}</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"both are zero"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    Point <span class="token punctuation">{</span> x<span class="token punctuation">:</span> <span class="token number">0</span><span class="token punctuation">,</span> y <span class="token punctuation">}</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"x is zero and y is {}"</span><span class="token punctuation">,</span> y<span class="token punctuation">)</span><span class="token punctuation">,</span>    Point <span class="token punctuation">{</span> x<span class="token punctuation">,</span> y<span class="token punctuation">:</span> <span class="token number">0</span> <span class="token punctuation">}</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"x is {} and y is zero"</span><span class="token punctuation">,</span> x<span class="token punctuation">)</span><span class="token punctuation">,</span>    Point <span class="token punctuation">{</span> x<span class="token punctuation">,</span> y <span class="token punctuation">}</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"x is {} and y is {}"</span><span class="token punctuation">,</span> x<span class="token punctuation">,</span> y<span class="token punctuation">)</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><p>It’s a bit concise compared to the <code>if else</code> logic but also might be confusing as we’re performing comparison, destructuring and variable binding at the same time.</p><p>This is how it looks like visually:</p><p><img src="/images/2020-rust-for-javascript-developers-4/pattern-matching-rust-1.png" alt=""><br><img src="/images/2020-rust-for-javascript-developers-4/pattern-matching-rust-2.png" alt=""></p><p>We begin to see why it’s named as “pattern matching” - we take an input and see which pattern in the match arms “fits” better - It’s like the <a href="https://www.google.com/search?q=shape+sorter" target="_blank" rel="noopener">shape sorter</a> toys that kids play with. Apart from comparison, we also do variable binding in the 2nd, 3rd and 4th match arms. We pass variables x or y or both to their respective expressions.</p><p>Pattern matching is also <code>exhaustive</code> - that is, it forces you to handle all the possible cases. Try removing the last match arm and Rust won’t let you compile the code.</p><h2 id="Enum"><a href="#Enum" class="headerlink" title="Enum"></a>Enum</h2><p>JavaScript doesn’t have Enums but if you’ve used TypeScript, you can think of Rust’s Enums as a combination of TypeScript’s <a href="https://www.typescriptlang.org/docs/handbook/enums.html" target="_blank" rel="noopener">Enums</a> and TypeScript’s <a href="https://www.typescriptlang.org/docs/handbook/advanced-types.html#discriminated-unions" target="_blank" rel="noopener">Discriminated Unions</a></p><p>In the simplest case, Enums can be used as a group of constants.</p><p>For example, even though JavaScript doesn’t have Enums, you might have used this pattern:</p><pre class=" language-javascript"><code class="language-javascript"><span class="token keyword">const</span> DIRECTION <span class="token operator">=</span> <span class="token punctuation">{</span>  FORWARD<span class="token punctuation">:</span> <span class="token string">"FORWARD"</span><span class="token punctuation">,</span>  BACKWARD<span class="token punctuation">:</span> <span class="token string">"BACKWARD"</span><span class="token punctuation">,</span>  LEFT<span class="token punctuation">:</span> <span class="token string">"LEFT"</span><span class="token punctuation">,</span>  RIGHT<span class="token punctuation">:</span> <span class="token string">"RIGHT"</span><span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token keyword">function</span> <span class="token function">move_drone</span><span class="token punctuation">(</span>direction<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">switch</span> <span class="token punctuation">(</span>direction<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">case</span> DIRECTION<span class="token punctuation">.</span>FORWARD<span class="token punctuation">:</span>      console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"Move Forward"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>      <span class="token keyword">break</span><span class="token punctuation">;</span>    <span class="token keyword">case</span> DIRECTION<span class="token punctuation">.</span>BACKWARD<span class="token punctuation">:</span>      console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"Move Backward"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>      <span class="token keyword">break</span><span class="token punctuation">;</span>    <span class="token keyword">case</span> DIRECTION<span class="token punctuation">.</span>LEFT<span class="token punctuation">:</span>      console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"Move Left"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>      <span class="token keyword">break</span><span class="token punctuation">;</span>    <span class="token keyword">case</span> DIRECTION<span class="token punctuation">.</span>RIGHT<span class="token punctuation">:</span>      console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"Move Right"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>      <span class="token keyword">break</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token function">move_drone</span><span class="token punctuation">(</span>DIRECTION<span class="token punctuation">.</span>FORWARD<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// "Move Forward"</span></code></pre><p>Here, we could’ve defined the FORWARD, BACKWARD, LEFT and RIGHT as separate constants, yet grouping it inside the DIRECTION object has the following benefits:</p><ul><li>The names FORWARD, BACKWARD, LEFT and RIGHT are namespaced under DIRECTION so naming conflicts can be avoided</li><li>It is self-documenting as we can quickly see all the valid directions available in the codebase</li></ul><p>However, there are some problems with this approach:</p><ul><li>What if someone passes NORTH or UP as an argument to move_drone? To fix this, we can add a validation to make sure that only values present in the DIRECTION object is allowed in the move function.</li><li>What if we decide to support UP and DOWN in future or rename LEFT/RIGHT to PORT/STARBOARD? We need to find all the places where similar switch-case or if-else is used. There’s a chance that we might miss out a few places which would cause issues in production.</li></ul><p>Enums in strongly typed languages like Rust are more powerful as they solve these problems without us writing extra code.</p><ul><li>If a function can take in only a small set of valid inputs, Enums can be used to enforce this constraint</li><li>Enums with pattern matching force you to cover all cases. Useful when you’re updating Enums in future</li></ul><p>Here’s the equivalent Rust example:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">enum</span> Direction <span class="token punctuation">{</span>  Forward<span class="token punctuation">,</span>  Backward<span class="token punctuation">,</span>  Left<span class="token punctuation">,</span>  Right<span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token keyword">fn</span> <span class="token function">move_drone</span><span class="token punctuation">(</span>direction<span class="token punctuation">:</span> Direction<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">match</span> direction <span class="token punctuation">{</span>    Direction<span class="token punctuation">:</span><span class="token punctuation">:</span>Forward <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"Move Forward"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    Direction<span class="token punctuation">:</span><span class="token punctuation">:</span>Backward <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"Move Backward"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    Direction<span class="token punctuation">:</span><span class="token punctuation">:</span>Left <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"Move Left"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    Direction<span class="token punctuation">:</span><span class="token punctuation">:</span>Right <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"Move Right"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token function">move_drone</span><span class="token punctuation">(</span>Direction<span class="token punctuation">:</span><span class="token punctuation">:</span>Forward<span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>We access the <code>variants</code> inside Enum using the <code>::</code> notation. Try editing this code by calling “move_drone(Direction::Up)” or adding “Down” as a new item in the Direction enum. In the first case, the compiler will throw an error saying that “Up” is not found in “Direction” and in the second case, the compiler will complain that we haven’t covered “Down” in the match block.</p><p>Rust Enums can do much more than act as a group of constants - we can also associate data with an Enum variant.</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">enum</span> Direction <span class="token punctuation">{</span>  Forward<span class="token punctuation">,</span>  Backward<span class="token punctuation">,</span>  Left<span class="token punctuation">,</span>  Right<span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token keyword">enum</span> Operation <span class="token punctuation">{</span>  PowerOn<span class="token punctuation">,</span>  PowerOff<span class="token punctuation">,</span>  <span class="token function">Move</span><span class="token punctuation">(</span>Direction<span class="token punctuation">)</span><span class="token punctuation">,</span>  Rotate<span class="token punctuation">,</span>  TakePhoto <span class="token punctuation">{</span> is_landscape<span class="token punctuation">:</span> bool<span class="token punctuation">,</span> zoom_level<span class="token punctuation">:</span> i32 <span class="token punctuation">}</span><span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token keyword">fn</span> <span class="token function">operate_drone</span><span class="token punctuation">(</span>operation<span class="token punctuation">:</span> Operation<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">match</span> operation <span class="token punctuation">{</span>    Operation<span class="token punctuation">:</span><span class="token punctuation">:</span>PowerOn <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"Power On"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    Operation<span class="token punctuation">:</span><span class="token punctuation">:</span>PowerOff <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"Power Off"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    Operation<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">Move</span><span class="token punctuation">(</span>direction<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token function">move_drone</span><span class="token punctuation">(</span>direction<span class="token punctuation">)</span><span class="token punctuation">,</span>    Operation<span class="token punctuation">:</span><span class="token punctuation">:</span>Rotate <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"Rotate"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    Operation<span class="token punctuation">:</span><span class="token punctuation">:</span>TakePhoto <span class="token punctuation">{</span>      is_landscape<span class="token punctuation">,</span>      zoom_level<span class="token punctuation">,</span>    <span class="token punctuation">}</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"TakePhoto {}, {}"</span><span class="token punctuation">,</span> is_landscape<span class="token punctuation">,</span> zoom_level<span class="token punctuation">)</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token keyword">fn</span> <span class="token function">move_drone</span><span class="token punctuation">(</span>direction<span class="token punctuation">:</span> Direction<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">match</span> direction <span class="token punctuation">{</span>    Direction<span class="token punctuation">:</span><span class="token punctuation">:</span>Forward <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"Move Forward"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    Direction<span class="token punctuation">:</span><span class="token punctuation">:</span>Backward <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"Move Backward"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    Direction<span class="token punctuation">:</span><span class="token punctuation">:</span>Left <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"Move Left"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    Direction<span class="token punctuation">:</span><span class="token punctuation">:</span>Right <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"Move Right"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token function">operate_drone</span><span class="token punctuation">(</span>Operation<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">Move</span><span class="token punctuation">(</span>Direction<span class="token punctuation">:</span><span class="token punctuation">:</span>Forward<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">operate_drone</span><span class="token punctuation">(</span>Operation<span class="token punctuation">:</span><span class="token punctuation">:</span>TakePhoto <span class="token punctuation">{</span>    is_landscape<span class="token punctuation">:</span> <span class="token keyword">true</span><span class="token punctuation">,</span>    zoom_level<span class="token punctuation">:</span> <span class="token number">10</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">}</span></code></pre><p>Here, we’ve added one more Enum called Operation that contains “unit like” variants (PowerOn, PowerOff, Rotate) and “struct like” variants (Move, TakePhoto). Notice how we’ve used pattern matching with destructuring and variable binding.</p><p>If you’ve used TypeScript or Flow, this is similar to <code>discriminated unions</code> or <code>sum types</code>:</p><pre class=" language-typescript"><code class="language-typescript"><span class="token keyword">interface</span> <span class="token class-name">PowerOn</span> <span class="token punctuation">{</span>  kind<span class="token punctuation">:</span> <span class="token string">"PowerOn"</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">interface</span> <span class="token class-name">PowerOff</span> <span class="token punctuation">{</span>  kind<span class="token punctuation">:</span> <span class="token string">"PowerOff"</span><span class="token punctuation">;</span><span class="token punctuation">}</span>type Direction <span class="token operator">=</span> <span class="token string">"Forward"</span> <span class="token operator">|</span> <span class="token string">"Backward"</span> <span class="token operator">|</span> <span class="token string">"Left"</span> <span class="token operator">|</span> <span class="token string">"Right"</span><span class="token punctuation">;</span><span class="token keyword">interface</span> <span class="token class-name">Move</span> <span class="token punctuation">{</span>  kind<span class="token punctuation">:</span> <span class="token string">"Move"</span><span class="token punctuation">;</span>  direction<span class="token punctuation">:</span> Direction<span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">interface</span> <span class="token class-name">Rotate</span> <span class="token punctuation">{</span>  kind<span class="token punctuation">:</span> <span class="token string">"Rotate"</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">interface</span> <span class="token class-name">TakePhoto</span> <span class="token punctuation">{</span>  kind<span class="token punctuation">:</span> <span class="token string">"TakePhoto"</span><span class="token punctuation">;</span>  is_landscape<span class="token punctuation">:</span> <span class="token keyword">boolean</span><span class="token punctuation">;</span>  zoom_level<span class="token punctuation">:</span> <span class="token keyword">number</span><span class="token punctuation">;</span><span class="token punctuation">}</span>type Operation <span class="token operator">=</span> PowerOn <span class="token operator">|</span> PowerOff <span class="token operator">|</span> Move <span class="token operator">|</span> Rotate <span class="token operator">|</span> TakePhoto<span class="token punctuation">;</span><span class="token keyword">function</span> <span class="token function">operate_drone</span><span class="token punctuation">(</span>operation<span class="token punctuation">:</span> Operation<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">switch</span> <span class="token punctuation">(</span>operation<span class="token punctuation">.</span>kind<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">case</span> <span class="token string">"PowerOn"</span><span class="token punctuation">:</span>      console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"Power On"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>      <span class="token keyword">break</span><span class="token punctuation">;</span>    <span class="token keyword">case</span> <span class="token string">"PowerOff"</span><span class="token punctuation">:</span>      console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"Power Off"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>      <span class="token keyword">break</span><span class="token punctuation">;</span>    <span class="token keyword">case</span> <span class="token string">"Move"</span><span class="token punctuation">:</span>      <span class="token function">move_drone</span><span class="token punctuation">(</span>operation<span class="token punctuation">.</span>direction<span class="token punctuation">)</span><span class="token punctuation">;</span>      <span class="token keyword">break</span><span class="token punctuation">;</span>    <span class="token keyword">case</span> <span class="token string">"Rotate"</span><span class="token punctuation">:</span>      console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"Rotate"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>      <span class="token keyword">break</span><span class="token punctuation">;</span>    <span class="token keyword">case</span> <span class="token string">"TakePhoto"</span><span class="token punctuation">:</span>      console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token string">`TakePhoto </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>operation<span class="token punctuation">.</span>is_landscape<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">, </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>operation<span class="token punctuation">.</span>zoom_level<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">;</span>      <span class="token keyword">break</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token keyword">function</span> <span class="token function">move_drone</span><span class="token punctuation">(</span>direction<span class="token punctuation">:</span> Direction<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">switch</span> <span class="token punctuation">(</span>direction<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">case</span> <span class="token string">"Forward"</span><span class="token punctuation">:</span>      console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"Move Forward"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>      <span class="token keyword">break</span><span class="token punctuation">;</span>    <span class="token keyword">case</span> <span class="token string">"Backward"</span><span class="token punctuation">:</span>      console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"Move Backward"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>      <span class="token keyword">break</span><span class="token punctuation">;</span>    <span class="token keyword">case</span> <span class="token string">"Left"</span><span class="token punctuation">:</span>      console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"Move Left"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>      <span class="token keyword">break</span><span class="token punctuation">;</span>    <span class="token keyword">case</span> <span class="token string">"Right"</span><span class="token punctuation">:</span>      console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"Move Right"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>      <span class="token keyword">break</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token function">operate_drone</span><span class="token punctuation">(</span><span class="token punctuation">{</span>  kind<span class="token punctuation">:</span> <span class="token string">"Move"</span><span class="token punctuation">,</span>  direction<span class="token punctuation">:</span> <span class="token string">"Forward"</span><span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token function">operate_drone</span><span class="token punctuation">(</span><span class="token punctuation">{</span>  kind<span class="token punctuation">:</span> <span class="token string">"TakePhoto"</span><span class="token punctuation">,</span>  is_landscape<span class="token punctuation">:</span> <span class="token keyword">true</span><span class="token punctuation">,</span>  zoom_level<span class="token punctuation">:</span> <span class="token number">10</span><span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><h2 id="Option"><a href="#Option" class="headerlink" title="Option"></a>Option</h2><p>We learnt about the <code>Option</code> type in <a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-variables-and-data-types/">chapter 2</a>. Option is actually an Enum with two variants - <code>Some</code> and <code>None</code>:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">enum</span> Option<span class="token operator">&lt;</span>T<span class="token operator">></span> <span class="token punctuation">{</span>  <span class="token function">Some</span><span class="token punctuation">(</span>T<span class="token punctuation">)</span><span class="token punctuation">,</span>  None<span class="token punctuation">,</span><span class="token punctuation">}</span></code></pre><p>This is how we handled the Option value in that chapter:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">fn</span> <span class="token function">read_file</span><span class="token punctuation">(</span>path<span class="token punctuation">:</span> <span class="token operator">&amp;</span>str<span class="token punctuation">)</span> <span class="token punctuation">-></span> Option<span class="token operator">&lt;</span><span class="token operator">&amp;</span>str<span class="token operator">></span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> contents <span class="token operator">=</span> <span class="token string">"hello"</span><span class="token punctuation">;</span>  <span class="token keyword">if</span> path <span class="token operator">!=</span> <span class="token string">""</span> <span class="token punctuation">{</span>    <span class="token keyword">return</span> <span class="token function">Some</span><span class="token punctuation">(</span>contents<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span>  <span class="token keyword">return</span> None<span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> file <span class="token operator">=</span> <span class="token function">read_file</span><span class="token punctuation">(</span><span class="token string">"path/to/file"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">if</span> file<span class="token punctuation">.</span><span class="token function">is_some</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">let</span> contents <span class="token operator">=</span> file<span class="token punctuation">.</span><span class="token function">unwrap</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> contents<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>    <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"Empty!"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><p>Using pattern matching, we can refactor the logic as follows:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> file <span class="token operator">=</span> <span class="token function">read_file</span><span class="token punctuation">(</span><span class="token string">"path/to/file"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">match</span> file <span class="token punctuation">{</span>    <span class="token function">Some</span><span class="token punctuation">(</span>contents<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> contents<span class="token punctuation">)</span><span class="token punctuation">,</span>    None <span class="token operator">=</span><span class="token operator">></span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"Empty!"</span><span class="token punctuation">)</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><p>Thanks for reading! Feel free to follow me in <a href="https://twitter.com/sheshbabu" target="_blank" rel="noopener">Twitter</a> for more posts like this :)</p>]]></content>
    
    <summary type="html">
    
      Part 4 covers pattern matching, destructuring, and enums
    
    </summary>
    
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
      <category term="Rust" scheme="https://www.sheshbabu.com/tags/Rust/"/>
    
      <category term="Rust Beginners" scheme="https://www.sheshbabu.com/tags/Rust-Beginners/"/>
    
      <category term="Rust for JavaScript Developers" scheme="https://www.sheshbabu.com/tags/Rust-for-JavaScript-Developers/"/>
    
  </entry>
  
  <entry>
    <title>Rust for JavaScript Developers - Functions and Control Flow</title>
    <link href="https://www.sheshbabu.com/posts/rust-for-javascript-developers-functions-and-control-flow/"/>
    <id>https://www.sheshbabu.com/posts/rust-for-javascript-developers-functions-and-control-flow/</id>
    <published>2020-07-05T13:18:00.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<p>This is the third part in a series about introducing the Rust language to JavaScript developers. Here are all the chapters:</p><ol><li><a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-tooling-ecosystem-overview/">Tooling Ecosystem Overview</a></li><li><a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-variables-and-data-types/">Variables and Data Types</a></li><li><a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-functions-and-control-flow/">Functions and Control Flow</a></li><li><a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-pattern-matching-and-enums/">Pattern Matching and Enums</a></li></ol><h2 id="Functions"><a href="#Functions" class="headerlink" title="Functions"></a>Functions</h2><p>Rust’s function syntax is pretty much similar to the one in JavaScript.</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> income <span class="token operator">=</span> <span class="token number">100</span><span class="token punctuation">;</span>  <span class="token keyword">let</span> tax <span class="token operator">=</span> <span class="token function">calculate_tax</span><span class="token punctuation">(</span>income<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> tax<span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">fn</span> <span class="token function">calculate_tax</span><span class="token punctuation">(</span>income<span class="token punctuation">:</span> i32<span class="token punctuation">)</span> <span class="token punctuation">-></span> i32 <span class="token punctuation">{</span>  <span class="token keyword">return</span> income <span class="token operator">*</span> <span class="token number">90</span> <span class="token operator">/</span> <span class="token number">100</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>The only difference you might see above is the type annotations for arguments and return values.</p><p>The <code>return</code> keyword can be skipped and it’s very common to see code without an explicit return. If you’re returning implicitly, make sure to remove the semicolon from that line. The above function can be refactored as:</p><pre class=" language-diff"><code class="language-diff">fn main() {  let income = 100;  let tax = calculate_tax(income);  println!("{}", tax);}fn calculate_tax(income: i32) -> i32 {<span class="token deleted">- return income * 90 / 100;</span><span class="token inserted">+ income * 90 / 100</span>}</code></pre><h2 id="Arrow-Functions"><a href="#Arrow-Functions" class="headerlink" title="Arrow Functions"></a>Arrow Functions</h2><p>Arrow functions are a popular feature in modern JavaScript - they allow us to write functional code in a concise way.</p><p>Rust has something similar and they are called “Closures”. The name might be a bit confusing and would require getting used to because in JavaScript, closures can be created using both normal and arrow functions.</p><p>Rust’s closure syntax is very similar to JavaScript’s arrow functions:</p><p><strong>Without arguments:</strong></p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// JavaScript</span><span class="token keyword">let</span> greet <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"hello"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token function">greet</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// "hello"</span></code></pre><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// Rust</span><span class="token keyword">let</span> greet <span class="token operator">=</span> <span class="token operator">||</span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"hello"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token function">greet</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// "hello"</span></code></pre><p><strong>With arguments:</strong></p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// JavaScript</span><span class="token keyword">let</span> greet <span class="token operator">=</span> <span class="token punctuation">(</span>msg<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>msg<span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token function">greet</span><span class="token punctuation">(</span><span class="token string">"good morning!"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// "good morning!"</span></code></pre><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// Rust</span><span class="token keyword">let</span> greet <span class="token operator">=</span> <span class="token operator">|</span>msg<span class="token punctuation">:</span> <span class="token operator">&amp;</span>str<span class="token operator">|</span> <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> msg<span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token function">greet</span><span class="token punctuation">(</span><span class="token string">"good morning!"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// "good morning!"</span></code></pre><p><strong>Returning values:</strong></p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// JavaScript</span><span class="token keyword">let</span> add <span class="token operator">=</span> <span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> a <span class="token operator">+</span> b<span class="token punctuation">;</span><span class="token function">add</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// 3</span></code></pre><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// Rust</span><span class="token keyword">let</span> add <span class="token operator">=</span> <span class="token closure-params"><span class="token punctuation">|</span>a<span class="token punctuation">:</span> i32<span class="token punctuation">,</span> b<span class="token punctuation">:</span> i32<span class="token punctuation">|</span></span> <span class="token punctuation">-></span> i32 <span class="token punctuation">{</span> a <span class="token operator">+</span> b <span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token function">add</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// 3</span></code></pre><p><strong>Multiline:</strong></p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// JavaScript</span><span class="token keyword">let</span> add <span class="token operator">=</span> <span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> sum <span class="token operator">=</span> a <span class="token operator">+</span> b<span class="token punctuation">;</span>  <span class="token keyword">return</span> sum<span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token function">add</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// 3</span></code></pre><pre class=" language-rust"><code class="language-rust"><span class="token comment" spellcheck="true">// Rust</span><span class="token keyword">let</span> add <span class="token operator">=</span> <span class="token closure-params"><span class="token punctuation">|</span>a<span class="token punctuation">:</span> i32<span class="token punctuation">,</span> b<span class="token punctuation">:</span> i32<span class="token punctuation">|</span></span> <span class="token punctuation">-></span> i32 <span class="token punctuation">{</span>  <span class="token keyword">let</span> sum <span class="token operator">=</span> a <span class="token operator">+</span> b<span class="token punctuation">;</span>  <span class="token keyword">return</span> sum<span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token function">add</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// 3</span></code></pre><p>Here’s a cheatsheet:<br><img src="/images/2020-rust-for-javascript-developers-3/image-2.png" alt=""></p><p>Closures don’t need the type annotations most of the time, but I’ve added them here for clarity.</p><h2 id="If-Else"><a href="#If-Else" class="headerlink" title="If Else"></a>If Else</h2><pre class=" language-rust"><code class="language-rust"><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> income <span class="token operator">=</span> <span class="token number">100</span><span class="token punctuation">;</span>  <span class="token keyword">let</span> tax <span class="token operator">=</span> <span class="token function">calculate_tax</span><span class="token punctuation">(</span>income<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> tax<span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">fn</span> <span class="token function">calculate_tax</span><span class="token punctuation">(</span>income<span class="token punctuation">:</span> i32<span class="token punctuation">)</span> <span class="token punctuation">-></span> i32 <span class="token punctuation">{</span>  <span class="token keyword">if</span> income <span class="token operator">&lt;</span> <span class="token number">10</span> <span class="token punctuation">{</span>    <span class="token keyword">return</span> <span class="token number">0</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token keyword">if</span> income <span class="token operator">>=</span> <span class="token number">10</span> <span class="token operator">&amp;&amp;</span> income <span class="token operator">&lt;</span> <span class="token number">50</span> <span class="token punctuation">{</span>    <span class="token keyword">return</span> <span class="token number">20</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>    <span class="token keyword">return</span> <span class="token number">50</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><h2 id="Loops"><a href="#Loops" class="headerlink" title="Loops"></a>Loops</h2><p>While loops:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> <span class="token keyword">mut</span> count <span class="token operator">=</span> <span class="token number">0</span><span class="token punctuation">;</span>  <span class="token keyword">while</span> count <span class="token operator">&lt;</span> <span class="token number">10</span> <span class="token punctuation">{</span>    <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> count<span class="token punctuation">)</span><span class="token punctuation">;</span>    count <span class="token operator">+=</span> <span class="token number">1</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><p>Normal <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for" target="_blank" rel="noopener">for loops</a> don’t exist in Rust, we need to use <code>while</code> or <code>for..in</code> loops. <code>for..in</code> loops are similar to the <code>for..of</code> loops in JavaScript and they loop over an iterator.</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> numbers <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">,</span> <span class="token number">5</span><span class="token punctuation">]</span><span class="token punctuation">;</span>  <span class="token keyword">for</span> n <span class="token keyword">in</span> numbers<span class="token punctuation">.</span><span class="token function">iter</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> n<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><p>Notice that we’re not iterating directly over the array but instead using the <code>iter</code> method of the array.</p><p>We can also loop over <a href="https://doc.rust-lang.org/reference/expressions/range-expr.html" target="_blank" rel="noopener">ranges</a>:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">for</span> n <span class="token keyword">in</span> <span class="token number">1</span><span class="token punctuation">..</span><span class="token number">5</span> <span class="token punctuation">{</span>    <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> n<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><h2 id="Iterators"><a href="#Iterators" class="headerlink" title="Iterators"></a>Iterators</h2><p>In JavaScript, we can use array methods like map/filter/reduce/etc instead of <code>for</code> loops to perform calculations or transformations on an array.</p><p>For example, here we take an array of numbers, double them and filter out the elements that are less than 10:</p><pre class=" language-javascript"><code class="language-javascript"><span class="token keyword">function</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> numbers <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">,</span> <span class="token number">5</span><span class="token punctuation">]</span><span class="token punctuation">;</span>  <span class="token keyword">let</span> double <span class="token operator">=</span> <span class="token punctuation">(</span>n<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> n <span class="token operator">*</span> <span class="token number">2</span><span class="token punctuation">;</span>  <span class="token keyword">let</span> less_than_ten <span class="token operator">=</span> <span class="token punctuation">(</span>n<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> n <span class="token operator">&lt;</span> <span class="token number">10</span><span class="token punctuation">;</span>  <span class="token keyword">let</span> result <span class="token operator">=</span> numbers<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span>double<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">filter</span><span class="token punctuation">(</span>less_than_ten<span class="token punctuation">)</span><span class="token punctuation">;</span>  console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>result<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// [2, 4, 6, 8]</span><span class="token punctuation">}</span></code></pre><p>In Rust, we can’t directly use map/filter/etc over vectors, we need to follow these steps:</p><ol><li>Convert the vector into an iterator using <code>iter</code>, <code>into_iter</code> or <code>iter_mut</code> methods</li><li>Chain <code>adapters</code> such as map/filter/etc on the iterator</li><li>Finally convert the iterator back to a vector using <code>consumers</code> such as <code>collect</code>, <code>find</code>, <code>sum</code> etc</li></ol><p>Here’s the equivalent Rust code:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> numbers <span class="token operator">=</span> <span class="token function">vec!</span><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">,</span> <span class="token number">5</span><span class="token punctuation">]</span><span class="token punctuation">;</span>  <span class="token keyword">let</span> double <span class="token operator">=</span> <span class="token closure-params"><span class="token punctuation">|</span>n<span class="token punctuation">:</span> <span class="token operator">&amp;</span>i32<span class="token punctuation">|</span></span> <span class="token punctuation">-></span> i32 <span class="token punctuation">{</span> n <span class="token operator">*</span> <span class="token number">2</span> <span class="token punctuation">}</span><span class="token punctuation">;</span>  <span class="token keyword">let</span> less_than_10 <span class="token operator">=</span> <span class="token closure-params"><span class="token punctuation">|</span>n<span class="token punctuation">:</span> <span class="token operator">&amp;</span>i32<span class="token punctuation">|</span></span> <span class="token punctuation">-></span> bool <span class="token punctuation">{</span> <span class="token operator">*</span>n <span class="token operator">&lt;</span> <span class="token number">10</span> <span class="token punctuation">}</span><span class="token punctuation">;</span>  <span class="token keyword">let</span> result<span class="token punctuation">:</span> Vec<span class="token operator">&lt;</span>i32<span class="token operator">></span> <span class="token operator">=</span> numbers<span class="token punctuation">.</span><span class="token function">iter</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span>double<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">filter</span><span class="token punctuation">(</span>less_than_10<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">collect</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{:?}"</span><span class="token punctuation">,</span> result<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// [2, 4, 6, 8]</span><span class="token punctuation">}</span></code></pre><p>You should be able to understand most of the code above but you might notice few things off here:</p><ul><li>The usage of <code>&amp;</code> and <code>*</code> in the closure</li><li>The <code>Vec&lt;i32&gt;</code> type annotation for the <code>result</code> variable</li></ul><p>The <code>&amp;</code> is the reference operator and the <code>*</code> is the dereference operator. The <code>iter</code> method instead of copying the elements in the vector, it passes them as references to the next adapter in the chain. This is why we use <code>&amp;i32</code> in the map’s closure (double). This closure returns <code>i32</code> but <a href="https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.filter" target="_blank" rel="noopener">filter</a> calls its closure (less_than_10) with reference so that’s why we need to use <code>&amp;i32</code> again. To dereference the argument, we use the <code>*</code> operator. We’ll cover this in more detail in future chapters.</p><p>Regarding <code>Vec&lt;i32&gt;</code>, so far we haven’t added type annotations to variables as Rust can infer the types automatically, but for <code>collect</code>, we need to be explicitly tell Rust that we expect a <code>Vec&lt;i32&gt;</code> output.</p><p>Aside from map and filter, there are ton of other <a href="https://doc.rust-lang.org/std/iter/trait.Iterator.html" target="_blank" rel="noopener">useful adapters</a> that we can use in iterators.</p><p>Thanks for reading! Feel free to follow me in <a href="https://twitter.com/sheshbabu" target="_blank" rel="noopener">Twitter</a> for more posts like this :)</p>]]></content>
    
    <summary type="html">
    
      Part 3 covers functions, closures, if-else, while, for..in, range and iterators
    
    </summary>
    
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
      <category term="Rust" scheme="https://www.sheshbabu.com/tags/Rust/"/>
    
      <category term="Rust Beginners" scheme="https://www.sheshbabu.com/tags/Rust-Beginners/"/>
    
      <category term="Rust for JavaScript Developers" scheme="https://www.sheshbabu.com/tags/Rust-for-JavaScript-Developers/"/>
    
  </entry>
  
  <entry>
    <title>Rust for JavaScript Developers - Variables and Data Types</title>
    <link href="https://www.sheshbabu.com/posts/rust-for-javascript-developers-variables-and-data-types/"/>
    <id>https://www.sheshbabu.com/posts/rust-for-javascript-developers-variables-and-data-types/</id>
    <published>2020-07-02T14:39:59.000Z</published>
    <updated>2024-05-19T06:17:28.272Z</updated>
    
    <content type="html"><![CDATA[<p>This is the second part in a series about introducing the Rust language to JavaScript developers. Here are all the chapters:</p><ol><li><a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-tooling-ecosystem-overview/">Tooling Ecosystem Overview</a></li><li><a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-variables-and-data-types/">Variables and Data Types</a></li><li><a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-functions-and-control-flow/">Functions and Control Flow</a></li><li><a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-pattern-matching-and-enums/">Pattern Matching and Enums</a></li></ol><h2 id="Variables"><a href="#Variables" class="headerlink" title="Variables"></a>Variables</h2><p>JavaScript has three ways to declare variables - <code>var</code>, <code>const</code> and <code>let</code>. Rust also has <code>const</code> and <code>let</code> but they work very differently compared to JavaScript.</p><h3 id="let"><a href="#let" class="headerlink" title="let"></a>let</h3><p>In Rust, we declare variables using <code>let</code>.</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">let</span> a <span class="token operator">=</span> <span class="token number">123</span><span class="token punctuation">;</span></code></pre><p>As expected, this assigns the value <code>123</code> to the variable <code>a</code>. But Rust by default makes the variable immutable. You won’t be able to change the value after it’s assigned.</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">let</span> a <span class="token operator">=</span> <span class="token number">123</span><span class="token punctuation">;</span>a <span class="token operator">=</span> <span class="token number">456</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// Error! :(</span></code></pre><p>At first glance, this might look similar to JavaScript’s const but JavaScript’s const doesn’t make variables immutable, it just makes the <a href="https://ponyfoo.com/articles/const-variables-not-immutable" target="_blank" rel="noopener">binding immutable</a>.</p><p>If you want to make variable mutable, you need to explicitly mention it using the <code>mut</code> keyword</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">let</span> <span class="token keyword">mut</span> a <span class="token operator">=</span> <span class="token number">123</span><span class="token punctuation">;</span>a <span class="token operator">=</span> <span class="token number">456</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// No Error :)</span></code></pre><h3 id="const"><a href="#const" class="headerlink" title="const"></a>const</h3><p>Rust’s <code>const</code> is also very different from JavaScript’s const - it’s better to think of Rust’s const as a “label” to a constant value. During compile time they get replaced by their actual values in all the places they are used. It’s usually used for constants like port numbers, timeout values, error codes etc.</p><p>You can also define const outside functions at global level which can’t be done using let.</p><h2 id="Data-Types"><a href="#Data-Types" class="headerlink" title="Data Types"></a>Data Types</h2><h3 id="Numbers"><a href="#Numbers" class="headerlink" title="Numbers"></a>Numbers</h3><p>In JavaScript, we’ve the <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number" target="_blank" rel="noopener">Number</a> type for both integers (numbers without decimal point) and floats (numbers with decimal point). In Rust, there’s a ton of options for <a href="https://doc.rust-lang.org/book/ch03-02-data-types.html#integer-types" target="_blank" rel="noopener">integers</a> and <a href="https://doc.rust-lang.org/book/ch03-02-data-types.html#floating-point-types" target="_blank" rel="noopener">floats</a> but by default we can use <code>i32</code> for integers and <code>f64</code> for floats.</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">let</span> x <span class="token operator">=</span> <span class="token number">123</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// i32</span><span class="token keyword">let</span> y <span class="token operator">=</span> <span class="token number">4.5</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// f64</span></code></pre><h3 id="Booleans"><a href="#Booleans" class="headerlink" title="Booleans"></a>Booleans</h3><p>Pretty straightforward - both JavaScript and Rust have booleans with true/false values</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">let</span> x <span class="token operator">=</span> <span class="token keyword">false</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// bool</span></code></pre><h3 id="Strings"><a href="#Strings" class="headerlink" title="Strings"></a>Strings</h3><p>We usually don’t think much about strings when working with JavaScript - they “just work”. In Rust, there are many types of strings but let’s focus on the widely used ones - <code>String</code> and <code>&amp;str</code>.</p><p><code>String</code> is growable whereas <code>&amp;str</code> is immutable and fixed size.</p><p>When you create a string using a string literal, it creates a <code>&amp;str</code> type:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">let</span> name <span class="token operator">=</span> <span class="token string">"Saitama"</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// &amp;str</span></code></pre><p>You need to use <code>String::from</code> or <code>to_string</code> methods to create a <code>String</code> type:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">let</span> name  <span class="token operator">=</span> String<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">from</span><span class="token punctuation">(</span><span class="token string">"Genos"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// String</span><span class="token keyword">let</span> name2 <span class="token operator">=</span> <span class="token string">"King"</span><span class="token punctuation">.</span><span class="token function">to_string</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token comment" spellcheck="true">// String</span></code></pre><p>You can convert from <code>String</code> to <code>&amp;str</code> using the <code>as_str</code> function</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">let</span> name2 <span class="token operator">=</span> <span class="token string">"King"</span><span class="token punctuation">.</span><span class="token function">to_string</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// String</span><span class="token keyword">let</span> name3 <span class="token operator">=</span> name2<span class="token punctuation">.</span><span class="token function">as_str</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>     <span class="token comment" spellcheck="true">// &amp;str</span></code></pre><p>We’ll learn more about strings in future posts.</p><h3 id="Optionals"><a href="#Optionals" class="headerlink" title="Optionals"></a>Optionals</h3><p>JavaScript has two types for empty values - <code>undefined</code> and <code>null</code>. Undefined is used when a variable, property etc is not defined and null is used when something is intentionally empty.</p><p>Rust has neither of these - it doesn’t even have a dedicated null data type. Instead it has something call <code>Option</code>. When we have a situation where a value can be empty or initially undefined, this <code>Option</code> type is used.</p><p>People who have worked with TypeScript/Flow might see some similarities here but it’s quite different in terms of how the optionals are created and used.</p><p>Say we want to write a function that takes in a file path and returns its contents. Let’s say for whatever reason, we want to return a “null” value when empty string is passed as file path.</p><p>Here’s how we would write this in JavaScript/TypeScript:</p><pre class=" language-typescript"><code class="language-typescript"><span class="token keyword">function</span> <span class="token function">read_file</span><span class="token punctuation">(</span>path<span class="token punctuation">:</span> <span class="token keyword">string</span><span class="token punctuation">)</span><span class="token punctuation">:</span> <span class="token keyword">string</span> <span class="token operator">|</span> <span class="token keyword">null</span> <span class="token punctuation">{</span>  <span class="token keyword">const</span> contents <span class="token operator">=</span> <span class="token string">"hello"</span><span class="token punctuation">;</span>  <span class="token keyword">if</span> <span class="token punctuation">(</span>path <span class="token operator">!==</span> <span class="token string">""</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">return</span> contents<span class="token punctuation">;</span>  <span class="token punctuation">}</span>  <span class="token keyword">return</span> <span class="token keyword">null</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>Implementing the same using Rust’s <code>Option</code>:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">fn</span> <span class="token function">read_file</span><span class="token punctuation">(</span>path<span class="token punctuation">:</span> <span class="token operator">&amp;</span>str<span class="token punctuation">)</span> <span class="token punctuation">-></span> Option<span class="token operator">&lt;</span><span class="token operator">&amp;</span>str<span class="token operator">></span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> contents <span class="token operator">=</span> <span class="token string">"hello"</span><span class="token punctuation">;</span>  <span class="token keyword">if</span> path <span class="token operator">!=</span> <span class="token string">""</span> <span class="token punctuation">{</span>    <span class="token keyword">return</span> <span class="token function">Some</span><span class="token punctuation">(</span>contents<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span>  <span class="token keyword">return</span> None<span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>You can see that we return <code>None</code> for null value but for non-null value, we don’t return the <code>contents</code> as it is, but rather we “wrap” it inside <code>Some</code> and return that. The return type is also not “string or null” as per the TypeScript example but “Option that contains &amp;str” type.</p><p>Here’s how you would call this function in JavaScript/TypeScript:</p><pre class=" language-javascript"><code class="language-javascript"><span class="token keyword">function</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">const</span> file_contents <span class="token operator">=</span> <span class="token function">read_file</span><span class="token punctuation">(</span><span class="token string">"/path/to/file"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">if</span> <span class="token punctuation">(</span>file_contents <span class="token operator">!==</span> <span class="token keyword">null</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>file_contents<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// file_contents is refined to string type</span>  <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>    console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"Empty!"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><p>Calling the function in Rust:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> file <span class="token operator">=</span> <span class="token function">read_file</span><span class="token punctuation">(</span><span class="token string">"path/to/file"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">if</span> file<span class="token punctuation">.</span><span class="token function">is_some</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">let</span> contents <span class="token operator">=</span> file<span class="token punctuation">.</span><span class="token function">unwrap</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{}"</span><span class="token punctuation">,</span> contents<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>    <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"Empty!"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><p>As you can see, we need to manually “unwrap” the Option to get the contents inside.</p><h3 id="Arrays"><a href="#Arrays" class="headerlink" title="Arrays"></a>Arrays</h3><p>Similar to strings, there are two types of arrays - one with fixed size (simply referred to as “Array”) and other that can grow/shrink in size (called “Vectors”).</p><p>Arrays:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> list <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">]</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{:?}"</span><span class="token punctuation">,</span> list<span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>Vectors:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> <span class="token keyword">mut</span> list <span class="token operator">=</span> <span class="token function">vec!</span><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">]</span><span class="token punctuation">;</span>  list<span class="token punctuation">.</span><span class="token function">push</span><span class="token punctuation">(</span><span class="token number">4</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{:?}"</span><span class="token punctuation">,</span> list<span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><h3 id="Objects"><a href="#Objects" class="headerlink" title="Objects"></a>Objects</h3><p>Technically, all non-primitive types are “objects” in JavaScript but we commonly use the term “object” for two things - bag of data or hash map.</p><p><strong>Bag of data:</strong><br>Unlike other languages, you need not go through a lot of ceremony to create an object and this is one of the coolest things about JavaScript.</p><p>To create an employee object in JavaScript:</p><pre class=" language-javascript"><code class="language-javascript"><span class="token keyword">function</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">const</span> employee <span class="token operator">=</span> <span class="token punctuation">{</span>    name<span class="token punctuation">:</span> <span class="token string">"Saitama"</span><span class="token punctuation">,</span>    age<span class="token punctuation">:</span> <span class="token number">25</span><span class="token punctuation">,</span>    occupation<span class="token punctuation">:</span> <span class="token string">"Hero"</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>To create the same object in Rust, we can use structs:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">struct</span> Employee <span class="token punctuation">{</span>  name<span class="token punctuation">:</span> String<span class="token punctuation">,</span>  age<span class="token punctuation">:</span> i32<span class="token punctuation">,</span>  occupation<span class="token punctuation">:</span> String<span class="token punctuation">,</span><span class="token punctuation">}</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> employee <span class="token operator">=</span> Employee <span class="token punctuation">{</span>    name<span class="token punctuation">:</span> <span class="token string">"Saitama"</span><span class="token punctuation">.</span><span class="token function">to_string</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>    age<span class="token punctuation">:</span> <span class="token number">25</span><span class="token punctuation">,</span>    occupation<span class="token punctuation">:</span> <span class="token string">"Hero"</span><span class="token punctuation">.</span><span class="token function">to_string</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>  <span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p><strong>HashMap:</strong></p><p>In JavaScript, to create an object with arbitrary key value pairs, we can either use normal object literals or the Map object:</p><pre class=" language-javascript"><code class="language-javascript"><span class="token keyword">function</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">const</span> colors <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">Map</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  colors<span class="token punctuation">.</span><span class="token keyword">set</span><span class="token punctuation">(</span><span class="token string">"white"</span><span class="token punctuation">,</span> <span class="token string">"#fff"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  colors<span class="token punctuation">.</span><span class="token keyword">set</span><span class="token punctuation">(</span><span class="token string">"black"</span><span class="token punctuation">,</span> <span class="token string">"#000"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>colors<span class="token punctuation">.</span><span class="token keyword">get</span><span class="token punctuation">(</span><span class="token string">"white"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// #fff</span><span class="token punctuation">}</span></code></pre><p>In Rust, you can do the same using the HashMap type:</p><pre class=" language-rust"><code class="language-rust"><span class="token keyword">use</span> std<span class="token punctuation">:</span><span class="token punctuation">:</span>collections<span class="token punctuation">:</span><span class="token punctuation">:</span>HashMap<span class="token punctuation">;</span><span class="token keyword">fn</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">let</span> <span class="token keyword">mut</span> colors <span class="token operator">=</span> HashMap<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">new</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  colors<span class="token punctuation">.</span><span class="token function">insert</span><span class="token punctuation">(</span><span class="token string">"white"</span><span class="token punctuation">.</span><span class="token function">to_string</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">"#fff"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  colors<span class="token punctuation">.</span><span class="token function">insert</span><span class="token punctuation">(</span><span class="token string">"black"</span><span class="token punctuation">.</span><span class="token function">to_string</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">"#000"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">println!</span><span class="token punctuation">(</span><span class="token string">"{:?}"</span><span class="token punctuation">,</span> colors<span class="token punctuation">.</span><span class="token function">get</span><span class="token punctuation">(</span><span class="token string">"white"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">unwrap</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// #fff</span><span class="token punctuation">}</span></code></pre><p>Notice the usage of unwrap above. HashMap’s get method returns an Option type which we need to unwrap to get the value inside.</p><p>Thanks for reading! Feel free to follow me in <a href="https://twitter.com/sheshbabu" target="_blank" rel="noopener">Twitter</a> for more posts like this :)</p>]]></content>
    
    <summary type="html">
    
      Rust for JavaScript Developers - Variables and Data Types
    
    </summary>
    
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
      <category term="Rust" scheme="https://www.sheshbabu.com/tags/Rust/"/>
    
      <category term="Rust Beginners" scheme="https://www.sheshbabu.com/tags/Rust-Beginners/"/>
    
      <category term="Rust for JavaScript Developers" scheme="https://www.sheshbabu.com/tags/Rust-for-JavaScript-Developers/"/>
    
  </entry>
  
  <entry>
    <title>Rust for JavaScript Developers - Tooling Ecosystem Overview</title>
    <link href="https://www.sheshbabu.com/posts/rust-for-javascript-developers-tooling-ecosystem-overview/"/>
    <id>https://www.sheshbabu.com/posts/rust-for-javascript-developers-tooling-ecosystem-overview/</id>
    <published>2020-06-28T13:35:17.000Z</published>
    <updated>2024-05-19T06:17:28.272Z</updated>
    
    <content type="html"><![CDATA[<p>This is the first part in a series about introducing the Rust language to JavaScript developers. Here are all the chapters:</p><ol><li><a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-tooling-ecosystem-overview/">Tooling Ecosystem Overview</a></li><li><a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-variables-and-data-types/">Variables and Data Types</a></li><li><a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-functions-and-control-flow/">Functions and Control Flow</a></li><li><a href="http://www.sheshbabu.com/posts/rust-for-javascript-developers-pattern-matching-and-enums/">Pattern Matching and Enums</a></li></ol><p>I find it easier to understand something new if it was explained in terms of something I already know - I thought there might be others like me :)</p><p>Here’s the tl;dr version:</p><table><thead><tr><th>Tool</th><th>JavaScript</th><th>Rust</th></tr></thead><tbody><tr><td>Version Manager</td><td>nvm</td><td>rustup</td></tr><tr><td>Package Manager</td><td>npm</td><td>Cargo</td></tr><tr><td>Package Registry</td><td>npmjs.com</td><td>crates.io</td></tr><tr><td>Package Manifest</td><td>package.json</td><td>Cargo.toml</td></tr><tr><td>Dependency Lockfile</td><td>package-lock.json</td><td>Cargo.lock</td></tr><tr><td>Task Runner</td><td>npm scripts, gulp etc</td><td>make, cargo-make</td></tr><tr><td>Live Reload</td><td>nodemon</td><td>cargo-watch</td></tr><tr><td>Linter</td><td>ESLint, TSLint, JSLint</td><td>Clippy</td></tr><tr><td>Formatter</td><td>Prettier</td><td>rustfmt</td></tr><tr><td>Dependency Vulnerability Checker</td><td>npm audit</td><td>cargo-audit</td></tr></tbody></table><h2 id="Setup"><a href="#Setup" class="headerlink" title="Setup"></a>Setup</h2><p>Rust is installed using the <a href="https://rustup.rs" target="_blank" rel="noopener">rustup</a> command. rustup is similar to <a href="https://github.com/nvm-sh/nvm" target="_blank" rel="noopener">nvm</a> in Node.js. You can use it to install and manage multiple versions of Rust and more.</p><h2 id="Cargo"><a href="#Cargo" class="headerlink" title="Cargo"></a>Cargo</h2><p>Installing Rust using rustup also installs Cargo similar to how installing Node.js also installs NPM. Cargo is Rust’s package manager and would feel very familiar if you’ve used NPM before.</p><p>Rust’s packages are called “crates”, and they’re downloaded from the <a href="https://crates.io" target="_blank" rel="noopener">crates.io</a> registry similar to how NPM packages are downloaded from <a href="https://www.npmjs.com" target="_blank" rel="noopener">npmjs.com</a>.</p><p>NPM while primarily a package manager, is also used as a task runner using the <a href="https://docs.npmjs.com/misc/scripts" target="_blank" rel="noopener">npm scripts</a> feature. Cargo has builtin support for common tasks like running code, building code etc. Cargo has features like <a href="https://doc.rust-lang.org/cargo/reference/workspaces.html" target="_blank" rel="noopener">workspaces</a> (similar to <a href="https://lerna.js.org" target="_blank" rel="noopener">lerna</a>), <a href="https://doc.rust-lang.org/cargo/reference/overriding-dependencies.html" target="_blank" rel="noopener">dependency overrides</a> (similar to <a href="https://www.npmjs.com/package/patch-package" target="_blank" rel="noopener">patch-package</a>) out of the box. It is also a test runner (similar to mocha, jest etc), benchmark runner etc.</p><p>So basically, Cargo is NPM on steroids!</p><h2 id="Project-Setup"><a href="#Project-Setup" class="headerlink" title="Project Setup"></a>Project Setup</h2><p>Creating a new project is done by running</p><pre class=" language-shell"><code class="language-shell">$ cargo new hello_rust</code></pre><p>This is somewhat similar to <code>npm init</code>. This creates a directory called “hello_rust” with the following files:</p><pre class=" language-shell"><code class="language-shell">hello_rust├── .git├── .gitignore├── Cargo.toml└─┬ src  └── main.rs</code></pre><h2 id="Cargo-toml"><a href="#Cargo-toml" class="headerlink" title="Cargo.toml"></a>Cargo.toml</h2><p>This is the package manifest file similar to package.json. The lock file (package-lock.json equivalent) is named Cargo.lock. Opening the Cargo.toml, you’ll see a familiar layout:</p><pre class=" language-toml"><code class="language-toml">[package]name = "hello_rust"version = "0.1.0"authors = ["sheshbabu"]edition = "2018"[dependencies]</code></pre><p>The <code>[package]</code> <a href="https://toml.io/en/v0.5.0#section-16" target="_blank" rel="noopener">table</a> contains metadata like the crate name, author, keywords etc. The <code>[dependencies]</code> table is similar to the <a href="https://docs.npmjs.com/files/package.json#dependencies" target="_blank" rel="noopener">dependencies object in package.json</a>. Cargo.toml also supports <code>[dev-dependencies]</code> similar to <a href="https://docs.npmjs.com/files/package.json#devdependencies" target="_blank" rel="noopener">devDependencies</a>.</p><h2 id="Dependency-Management"><a href="#Dependency-Management" class="headerlink" title="Dependency Management"></a>Dependency Management</h2><p>Installing a new dependency is done by manually editing the Cargo.toml file, adding the dependency under <code>[dependencies]</code> and running <code>cargo build</code>. For example, if we want to install the “serde” crate, we need to edit the Cargo.toml file as follows:</p><pre class=" language-diff"><code class="language-diff">[package]name = "hello_rust"version = "0.1.0"authors = ["sheshbabu"]edition = "2018"[dependencies]<span class="token inserted">+ serde = "1.0.106"</span></code></pre><p>and run</p><pre class=" language-shell"><code class="language-shell">$ cargo build</code></pre><p>Similarly, to remove or update a dependency, we need to manually edit the Cargo.toml file and run <code>cargo build</code>. I was initially confused by the existence of the <code>cargo install</code> command but it turned out to be an equivalent of <code>npm install -g</code>.</p><p>If you want something similar to <code>npm install</code>, <code>npm update</code> or <code>npm uninstall</code>, you can install <a href="https://crates.io/crates/cargo-edit" target="_blank" rel="noopener">cargo-edit</a> which enhances Cargo with <code>cargo add</code>, <code>cargo rm</code> and <code>cargo upgrade</code> subcommands.</p><p>You can also specify the <a href="https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#caret-requirements" target="_blank" rel="noopener">dependency version using patterns</a> similar to NPM.</p><h2 id="Development-tools"><a href="#Development-tools" class="headerlink" title="Development tools"></a>Development tools</h2><h3 id="Task-Runner"><a href="#Task-Runner" class="headerlink" title="Task Runner"></a>Task Runner</h3><p>Cargo supports running common tasks like build, run, test etc. But if you want something similar to NPM scripts, you can use make or <a href="https://crates.io/crates/cargo-make" target="_blank" rel="noopener">cargo-make</a></p><h3 id="Live-Reload"><a href="#Live-Reload" class="headerlink" title="Live Reload"></a>Live Reload</h3><p>Nodemon is an essential tool for Node.js development - it watches for changes to files and automatically restarts the application. <a href="https://crates.io/crates/cargo-watch" target="_blank" rel="noopener">cargo-watch</a> is the equivalent in Rust world.</p><h3 id="Linter-and-Formatter"><a href="#Linter-and-Formatter" class="headerlink" title="Linter and Formatter"></a>Linter and Formatter</h3><p>Rust has builtin linter called <a href="https://github.com/rust-lang/rust-clippy" target="_blank" rel="noopener">Clippy</a> and formatter called <a href="https://github.com/rust-lang/rustfmt" target="_blank" rel="noopener">rustfmt</a>. They’re equivalent to ESLint and Prettier in JS ecosystem. Precommit hooks can be managed using <a href="https://crates.io/crates/cargo-husky" target="_blank" rel="noopener">cargo-husky</a>.</p><h3 id="Vulnerability-Checking"><a href="#Vulnerability-Checking" class="headerlink" title="Vulnerability Checking"></a>Vulnerability Checking</h3><p>Scanning for vulnerabilities in dependencies is done using <a href="https://crates.io/crates/cargo-audit" target="_blank" rel="noopener">cargo-audit</a> and is very similar to <a href="https://docs.npmjs.com/cli/audit" target="_blank" rel="noopener">npm audit</a>.</p><p>Thanks for reading! Feel free to follow me in <a href="https://twitter.com/sheshbabu" target="_blank" rel="noopener">Twitter</a> for more posts like this :)</p>]]></content>
    
    <summary type="html">
    
      Rust for JavaScript Developers - Tooling Ecosystem Overview
    
    </summary>
    
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
      <category term="Rust" scheme="https://www.sheshbabu.com/tags/Rust/"/>
    
      <category term="Rust Beginners" scheme="https://www.sheshbabu.com/tags/Rust-Beginners/"/>
    
      <category term="Rust for JavaScript Developers" scheme="https://www.sheshbabu.com/tags/Rust-for-JavaScript-Developers/"/>
    
  </entry>
  
  <entry>
    <title>Automatic PageView Tracking using React Router</title>
    <link href="https://www.sheshbabu.com/posts/automatic-pageview-tracking-using-react-router/"/>
    <id>https://www.sheshbabu.com/posts/automatic-pageview-tracking-using-react-router/</id>
    <published>2020-06-20T06:27:51.000Z</published>
    <updated>2024-05-19T06:17:28.270Z</updated>
    
    <content type="html"><![CDATA[<p>Analytics providers like Google Analytics, Amplitude etc work really well with server rendered pages for tracking pageviews but fall short when you want to use them in Single Page Applications (SPA).</p><p>You would need to manually call their pageview tracking apis on every navigation.</p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// Google Analytics</span><span class="token function">ga</span><span class="token punctuation">(</span><span class="token string">"set"</span><span class="token punctuation">,</span> <span class="token string">"page"</span><span class="token punctuation">,</span> <span class="token string">"/account"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token function">ga</span><span class="token punctuation">(</span><span class="token string">"send"</span><span class="token punctuation">,</span> <span class="token string">"pageview"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token comment" spellcheck="true">// Amplitude</span>amplitude<span class="token punctuation">.</span><span class="token function">logEvent</span><span class="token punctuation">(</span><span class="token string">"PageView"</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> path<span class="token punctuation">:</span> <span class="token string">"/account"</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token comment" spellcheck="true">// Freshlytics</span>window<span class="token punctuation">.</span>__freshlytics__<span class="token punctuation">.</span><span class="token function">sendPageView</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>For example, let’s take a simple React app that has 3 routes:</p><pre class=" language-javascript"><code class="language-javascript"><span class="token keyword">import</span> React <span class="token keyword">from</span> <span class="token string">"react"</span><span class="token punctuation">;</span><span class="token keyword">import</span> <span class="token punctuation">{</span> BrowserRouter<span class="token punctuation">,</span> Switch<span class="token punctuation">,</span> Route<span class="token punctuation">,</span> Link <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"react-router-dom"</span><span class="token punctuation">;</span><span class="token keyword">export</span> <span class="token keyword">default</span> <span class="token keyword">function</span> <span class="token function">App</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">return</span> <span class="token punctuation">(</span>    <span class="token operator">&lt;</span>BrowserRouter<span class="token operator">></span>      <span class="token operator">&lt;</span>NavBar <span class="token operator">/</span><span class="token operator">></span>      <span class="token operator">&lt;</span>Routes <span class="token operator">/</span><span class="token operator">></span>    <span class="token operator">&lt;</span><span class="token operator">/</span>BrowserRouter<span class="token operator">></span>  <span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">function</span> <span class="token function">NavBar</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">return</span> <span class="token punctuation">(</span>    <span class="token operator">&lt;</span>nav<span class="token operator">></span>      <span class="token operator">&lt;</span>Link to<span class="token operator">=</span><span class="token string">"/"</span><span class="token operator">></span>Home<span class="token operator">&lt;</span><span class="token operator">/</span>Link<span class="token operator">></span>      <span class="token operator">&lt;</span>Link to<span class="token operator">=</span><span class="token string">"/users"</span><span class="token operator">></span>Users<span class="token operator">&lt;</span><span class="token operator">/</span>Link<span class="token operator">></span>      <span class="token operator">&lt;</span>Link to<span class="token operator">=</span><span class="token string">"/about"</span><span class="token operator">></span>About<span class="token operator">&lt;</span><span class="token operator">/</span>Link<span class="token operator">></span>    <span class="token operator">&lt;</span><span class="token operator">/</span>nav<span class="token operator">></span>  <span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">function</span> <span class="token function">Routes</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">return</span> <span class="token punctuation">(</span>    <span class="token operator">&lt;</span>Switch<span class="token operator">></span>      <span class="token operator">&lt;</span>Route exact path<span class="token operator">=</span><span class="token string">"/"</span> component<span class="token operator">=</span><span class="token punctuation">{</span>HomePage<span class="token punctuation">}</span> <span class="token operator">/</span><span class="token operator">></span>      <span class="token operator">&lt;</span>Route exact path<span class="token operator">=</span><span class="token string">"/users"</span> component<span class="token operator">=</span><span class="token punctuation">{</span>UsersPage<span class="token punctuation">}</span> <span class="token operator">/</span><span class="token operator">></span>      <span class="token operator">&lt;</span>Route exact path<span class="token operator">=</span><span class="token string">"/about"</span> component<span class="token operator">=</span><span class="token punctuation">{</span>AboutPage<span class="token punctuation">}</span> <span class="token operator">/</span><span class="token operator">></span>    <span class="token operator">&lt;</span><span class="token operator">/</span>Switch<span class="token operator">></span>  <span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">function</span> <span class="token function">HomePage</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">return</span> <span class="token operator">&lt;</span>h2<span class="token operator">></span>Home<span class="token operator">&lt;</span><span class="token operator">/</span>h2<span class="token operator">></span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">function</span> <span class="token function">UsersPage</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">return</span> <span class="token operator">&lt;</span>h2<span class="token operator">></span>Users<span class="token operator">&lt;</span><span class="token operator">/</span>h2<span class="token operator">></span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">function</span> <span class="token function">AboutPage</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">return</span> <span class="token operator">&lt;</span>h2<span class="token operator">></span>About<span class="token operator">&lt;</span><span class="token operator">/</span>h2<span class="token operator">></span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>To track the pageviews, we need to call the tracking api during <a href="https://reactjs.org/docs/react-component.html#mounting" target="_blank" rel="noopener">mount</a> in each of these pages</p><pre class=" language-diff"><code class="language-diff">function HomePage() {<span class="token inserted">+ React.useEffect(() => {</span><span class="token inserted">+   ga("set", "page", "/");</span><span class="token inserted">+   ga("send", "pageview");</span><span class="token inserted">+ }, []);</span>  return &lt;h2>Home&lt;/h2>;}function UsersPage() {<span class="token inserted">+ React.useEffect(() => {</span><span class="token inserted">+   ga("set", "page", "/users");</span><span class="token inserted">+   ga("send", "pageview");</span><span class="token inserted">+ }, []);</span>  return &lt;h2>Users&lt;/h2>;}function AboutPage() {<span class="token inserted">+ React.useEffect(() => {</span><span class="token inserted">+   ga("set", "page", "/about");</span><span class="token inserted">+   ga("send", "pageview");</span><span class="token inserted">+ }, []);</span>  return &lt;h2>About&lt;/h2>;}</code></pre><p>Not terrible, but there are some problems:</p><ul><li>The url paths (“/“, “/users”, “/about”) should be kept in sync with the router</li><li>The same code is repeated in multiple page components</li><li>Developers might forget to add this when creating new pages</li></ul><p>To avoid hardcoding the url paths, we can use the <code>window.location.pathname</code> api</p><pre class=" language-diff"><code class="language-diff">function HomePage() {  React.useEffect(() => {<span class="token deleted">-   ga("set", "page", "/");</span><span class="token inserted">+   ga("set", "page", window.location.pathname);</span>    ga("send", "pageview");  }, []);  return &lt;h2>Home&lt;/h2>;}function UsersPage() {  React.useEffect(() => {<span class="token deleted">-   ga("set", "page", "/users");</span><span class="token inserted">+   ga("set", "page", window.location.pathname);</span>    ga("send", "pageview");  }, []);  return &lt;h2>Users&lt;/h2>;}function AboutPage() {  React.useEffect(() => {<span class="token deleted">-   ga("set", "page", "/about");</span><span class="token inserted">+   ga("set", "page", window.location.pathname);</span>    ga("send", "pageview");  }, []);  return &lt;h2>About&lt;/h2>;}</code></pre><p>To fix code duplication and centralize the pageview tracking, we can use the <code>history</code> object and listen to it for navigation events. When navigation occurs, we can send the pageview event.</p><pre class=" language-diff"><code class="language-diff">import React from "react";<span class="token deleted">- import { BrowserRouter, Switch, Route, Link } from "react-router-dom";</span><span class="token inserted">+ import { BrowserRouter, Switch, Route, Link, useHistory } from "react-router-dom";</span>export default function App() {  return (    &lt;BrowserRouter>      &lt;NavBar />      &lt;Routes />    &lt;/BrowserRouter>  );}function NavBar() {  return (    &lt;nav>      &lt;Link to="/">Home&lt;/Link>      &lt;Link to="/users">Users&lt;/Link>      &lt;Link to="/about">About&lt;/Link>    &lt;/nav>  );}function Routes() {<span class="token inserted">+  const history = useHistory();</span><span class="token inserted">+</span><span class="token inserted">+  React.useEffect(() => {</span><span class="token inserted">+    trackPageView(); // To track the first pageview upon load</span><span class="token inserted">+    history.listen(trackPageView); // To track the subsequent pageviews</span><span class="token inserted">+  }, [history]);</span><span class="token inserted">+</span><span class="token inserted">+  function trackPageView() {</span><span class="token inserted">+    ga("set", "page", window.location.pathname);</span><span class="token inserted">+    ga("send", "pageview");</span><span class="token inserted">+  }</span>  return (    &lt;Switch>      &lt;Route exact path="/" component={HomePage} />      &lt;Route exact path="/users" component={UsersPage} />      &lt;Route exact path="/about" component={AboutPage} />    &lt;/Switch>  );}function HomePage() {<span class="token deleted">-  React.useEffect(() => {</span><span class="token deleted">-    ga("set", "page", window.location.pathname);</span><span class="token deleted">-    ga("send", "pageview");</span><span class="token deleted">-  }, []);</span>  return &lt;h2>Home&lt;/h2>;}function UsersPage() {<span class="token deleted">-  React.useEffect(() => {</span><span class="token deleted">-    ga("set", "page", window.location.pathname);</span><span class="token deleted">-    ga("send", "pageview");</span><span class="token deleted">-  }, []);</span>  return &lt;h2>Users&lt;/h2>;}function AboutPage() {<span class="token deleted">-  React.useEffect(() => {</span><span class="token deleted">-    ga("set", "page", window.location.pathname);</span><span class="token deleted">-    ga("send", "pageview");</span><span class="token deleted">-  }, []);</span>  return &lt;h2>About&lt;/h2>;}</code></pre><p>Now the pageviews for all routes can be tracked automatically without adding any tracking code in the page components! We can refine this further and extract the tracking logic in <code>Routes</code> component to a <a href="https://reactjs.org/docs/hooks-custom.html" target="_blank" rel="noopener">custom hook</a>.</p><p>You can find a complete example in <a href="https://github.com/sheshbabu/react-pageview-tracking-demo" target="_blank" rel="noopener">this repo</a> and play with the <a href="https://react-pageview-tracking-demo.netlify.app" target="_blank" rel="noopener">demo here</a>.</p><p>Note: The examples in this post use React Router 5.2.0</p>]]></content>
    
    <summary type="html">
    
      Automatic PageView Tracking using React Router
    
    </summary>
    
    
      <category term="React" scheme="https://www.sheshbabu.com/tags/React/"/>
    
      <category term="Analytics" scheme="https://www.sheshbabu.com/tags/Analytics/"/>
    
  </entry>
  
  <entry>
    <title>Publishing an npx command to npm</title>
    <link href="https://www.sheshbabu.com/posts/publishing-npx-command-to-npm/"/>
    <id>https://www.sheshbabu.com/posts/publishing-npx-command-to-npm/</id>
    <published>2020-06-19T14:13:45.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<p><a href="https://blog.npmjs.org/post/162869356040/introducing-npx-an-npm-package-runner" target="_blank" rel="noopener">npx</a> is an useful tool to run one-off commands like <code>create-react-app</code>, <code>http-server</code> etc.</p><p>In this post I’ll go through the steps needed to create a command line tool that can be invoked using npx.</p><p>Let’s start with a simple hello world script</p><pre class=" language-javascript"><code class="language-javascript">console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"hello world"</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>This prints “hello world” when you run it</p><pre class=" language-shell"><code class="language-shell">$ node index.jshello world</code></pre><p>To make this runnable by npx, we need to convert this into an executable file by adding the node <a href="https://en.wikipedia.org/wiki/Shebang_(Unix)" target="_blank" rel="noopener">shebang</a> line</p><pre class=" language-javascript"><code class="language-javascript">#<span class="token operator">!</span><span class="token operator">/</span>usr<span class="token operator">/</span>bin<span class="token operator">/</span>env nodeconsole<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"hello world"</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>Let’s then create a npm package so this file can be published to npm</p><pre class=" language-shell"><code class="language-shell">$ npm init -y</code></pre><p>This will create the package.json file.</p><p>If you publish this package now, it can be installed by others but they won’t be able to execute it using npx. For this we need to reference this file in the <a href="https://docs.npmjs.com/files/package.json#bin" target="_blank" rel="noopener">bin</a> field of package.json</p><pre class=" language-json"><code class="language-json"><span class="token punctuation">{</span>  <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"npx-example"</span><span class="token punctuation">,</span>  <span class="token property">"version"</span><span class="token operator">:</span> <span class="token string">"1.0.0"</span><span class="token punctuation">,</span>  <span class="token property">"main"</span><span class="token operator">:</span> <span class="token string">"index.js"</span><span class="token punctuation">,</span>  <span class="token property">"bin"</span><span class="token operator">:</span> <span class="token string">"index.js"</span><span class="token punctuation">,</span>  <span class="token property">"scripts"</span><span class="token operator">:</span> <span class="token punctuation">{</span>    <span class="token property">"start"</span><span class="token operator">:</span> <span class="token string">"node ."</span>  <span class="token punctuation">}</span><span class="token punctuation">,</span>  <span class="token property">"license"</span><span class="token operator">:</span> <span class="token string">"MIT"</span><span class="token punctuation">}</span></code></pre><p>You can now publish this package</p><pre class=" language-shell"><code class="language-shell">$ npm publish</code></pre><p>You can find a complete example in <a href="https://github.com/sheshbabu/gitrmbr" target="_blank" rel="noopener">this repo</a>.</p>]]></content>
    
    <summary type="html">
    
      Publishing an npx command to npm
    
    </summary>
    
    
      <category term="Node.js" scheme="https://www.sheshbabu.com/tags/Node-js/"/>
    
  </entry>
  
  <entry>
    <title>Running Express over HTTPS in localhost</title>
    <link href="https://www.sheshbabu.com/posts/running-express-over-https-in-localhost/"/>
    <id>https://www.sheshbabu.com/posts/running-express-over-https-in-localhost/</id>
    <published>2020-06-19T11:29:14.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<p>There are times when you want to expose your Express server via HTTPS for local development.</p><p>Let’s take a super simple Express code:</p><pre class=" language-javascript"><code class="language-javascript"><span class="token keyword">const</span> express <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"express"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token keyword">const</span> app <span class="token operator">=</span> <span class="token function">express</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>app<span class="token punctuation">.</span><span class="token keyword">get</span><span class="token punctuation">(</span><span class="token string">"/"</span><span class="token punctuation">,</span> <span class="token punctuation">(</span>req<span class="token punctuation">,</span> res<span class="token punctuation">,</span> next<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span>  res<span class="token punctuation">.</span><span class="token function">send</span><span class="token punctuation">(</span><span class="token string">"Hello World\n"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>app<span class="token punctuation">.</span><span class="token function">listen</span><span class="token punctuation">(</span><span class="token number">3001</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>This will return “Hello World” when you hit <a href="http://localhost:3001" target="_blank" rel="noopener">http://localhost:3001</a></p><pre class=" language-shell"><code class="language-shell">$ curl http://localhost:3001Hello World</code></pre><p>To run this in HTTPS, we need to install a tool called <a href="https://github.com/FiloSottile/mkcert" target="_blank" rel="noopener">mkcert</a>. Follow the <a href="https://github.com/FiloSottile/mkcert/blob/v1.4.1/README.md#installation" target="_blank" rel="noopener">installation instructions</a> or if you’re using macOS and Homebrew, run this command:</p><pre class=" language-shell"><code class="language-shell">$ brew install mkcert</code></pre><p>Create a local <a href="https://en.wikipedia.org/wiki/Certificate_authority" target="_blank" rel="noopener">CA</a> and create certificate for localhost:</p><pre class=" language-shell"><code class="language-shell">$ mkcert -install$ mkcert localhost</code></pre><p>This will create a certificate file <code>localhost.pem</code> and key file <code>localhost-key.pem</code> in the current directory.</p><p>Update the Express code as follows:</p><pre class=" language-diff"><code class="language-diff"><span class="token inserted">+ const fs = require("fs");</span><span class="token inserted">+ const https = require("https");</span>const express = require("express");<span class="token inserted">+ const key = fs.readFileSync("localhost-key.pem", "utf-8");</span><span class="token inserted">+ const cert = fs.readFileSync("localhost.pem", "utf-8");</span>const app = express();app.get("/", (req, res, next) => {  res.send("Hello World\n");});<span class="token deleted">- app.listen(3001);</span><span class="token inserted">+ https.createServer({ key, cert }, app).listen(3001);</span></code></pre><p>And you’re done!</p><pre class=" language-shell"><code class="language-shell">$ curl https://localhost:3001Hello World</code></pre><p>You can find the full codebase in <a href="https://github.com/sheshbabu/express-https-localhost" target="_blank" rel="noopener">this repo</a>.</p>]]></content>
    
    <summary type="html">
    
      Running Express over HTTPS in localhost
    
    </summary>
    
    
      <category term="Node.js" scheme="https://www.sheshbabu.com/tags/Node-js/"/>
    
      <category term="Express.js" scheme="https://www.sheshbabu.com/tags/Express-js/"/>
    
  </entry>
  
  <entry>
    <title>Visual explanation of SAML authentication</title>
    <link href="https://www.sheshbabu.com/posts/visual-explanation-of-saml-authentication/"/>
    <id>https://www.sheshbabu.com/posts/visual-explanation-of-saml-authentication/</id>
    <published>2020-06-18T23:31:25.000Z</published>
    <updated>2024-08-12T13:43:12.150Z</updated>
    
    <content type="html"><![CDATA[<p>SAML (Security Assertion Markup Language) is the most commonly used authentication protocol and SSO solution in enterprises.</p><p><img src="/images/2020-saml/image-1.png" alt=""></p><h2 id="What-is-SSO"><a href="#What-is-SSO" class="headerlink" title="What is SSO?"></a>What is SSO?</h2><p>To put it simply, it’s the enterprise equivalent of the “Login with Google” or “Login with Facebook” buttons we see in apps around the internet. We register an account initially in Google or Facebook etc and use that account to login to other apps like Spotify, Netflix, Zoom etc. We do this to avoid maintaining multiple username/passwords. Similarly, enterprises maintain a single user management system and employees use their corporate account to login to third-party services like Salesforce, Workday, Expensify etc without creating separate accounts or remembering multiple passwords. This is called SSO (Single Sign On) and SAML is the de facto enterprise SSO solution.</p><h2 id="Participants"><a href="#Participants" class="headerlink" title="Participants"></a>Participants</h2><p>There are 3 main participants involved in the SAML authentication flow:</p><h3 id="Identity-Provider-IdP"><a href="#Identity-Provider-IdP" class="headerlink" title="Identity Provider (IdP)"></a>Identity Provider (IdP)</h3><p>This is the centralised user management system that we talked about earlier. This server is responsible for authenticating the user and passing the user details such as email address, name, department etc to the Service Provider. Popular identity providers are Azure AD, Auth0, Onelogin, Okta, G Suite etc.</p><h3 id="Service-Provider-SP"><a href="#Service-Provider-SP" class="headerlink" title="Service Provider (SP)"></a>Service Provider (SP)</h3><p>This is the application that trusts the IdP and wants to use it for authentication. Examples: Salesforce, Workday, Expensify, $YOUR_AWESOME_APP etc</p><h3 id="Principal"><a href="#Principal" class="headerlink" title="Principal"></a>Principal</h3><p>This is the user who’s trying to log into the SP via the IdP.</p><h2 id="Authentication-Flows"><a href="#Authentication-Flows" class="headerlink" title="Authentication Flows"></a>Authentication Flows</h2><p>There are two common ways for an user to access SP:</p><h3 id="IdP-initiated-login"><a href="#IdP-initiated-login" class="headerlink" title="IdP initiated login:"></a>IdP initiated login:</h3><p>The user goes to the IdP first and is shown a list of SP they have access to. Upon choosing an SP from that list, they’re redirected to that SP.</p><h3 id="SP-initiated-login"><a href="#SP-initiated-login" class="headerlink" title="SP initiated login:"></a>SP initiated login:</h3><p>In this flow, the user goes to the SP’s website first. If the user doesn’t have an active session with the SP, the user is redirected to the IdP for authentication. Upon successful login, the user is redirected back to the SP. We’ll be discussing this flow in detail.</p><h2 id="SP-initiated-Flow"><a href="#SP-initiated-Flow" class="headerlink" title="SP initiated Flow:"></a>SP initiated Flow:</h2><p>Let’s talk about the flow from the user’s perspective.</p><ul><li>The user goes to the SP’s website. If the user is not logged in, it shows a “Login with SSO” button</li><li>Upon clicking the login button, the user is redirected to the IdP’s website where they’re asked to submit their credentials</li><li>Upon successful login, the user is redirected back to the SP’s website where they can perform their work</li></ul><p><img src="/images/2020-saml/image-1.png" alt=""></p><p>Now, let’s zoom in a bit and understand what happens behind the scenes:</p><ul><li>SP checks for active session</li><li>SP sends AuthnRequest to IdP</li><li>IdP authenticates the user</li><li>IdP sends SAML Assertion to SP</li><li>SP creates session and logs in user</li></ul><h3 id="SP-checks-for-active-session"><a href="#SP-checks-for-active-session" class="headerlink" title="SP checks for active session"></a>SP checks for active session</h3><p>SAML doesn’t maintain sessions, so SP needs to maintain sessions for each authenticated user. When a user visits the SP website, it checks whether the user has an active session with it. If an active session exists, the user can enter the website otherwise a “Login with SSO” button is shown.</p><p><img src="/images/2020-saml/image-2.png" alt=""></p><h3 id="SP-sends-AuthRequest-to-IdP"><a href="#SP-sends-AuthRequest-to-IdP" class="headerlink" title="SP sends AuthRequest to IdP"></a>SP sends AuthRequest to IdP</h3><p>When the user clicks on the “Login with SSO” button, the SP generates a XML message called “AuthnRequest” with details about who’s sending the request (Issuer), where to redirect to after the user is authenticated (Assertion Consumer Service url) and security measures (ID, IssueInstant). Here’s an <a href="https://www.samltool.com/generic_sso_req.php" target="_blank" rel="noopener">example AuthnRequest XML</a>.</p><p>This XML is encoded into a url-safe string, embedded as query param in a request to IdP and the user is redirected to this IdP url:</p><p><code>https://idp.com/SAML2/SSO/Redirect?SAMLRequest=EncodedAuthnRequest</code></p><h3 id="IdP-authenticates-the-user"><a href="#IdP-authenticates-the-user" class="headerlink" title="IdP authenticates the user"></a>IdP authenticates the user</h3><p>IdP maintains its own session about the user and if an active session exists for the user, the user is redirected to SP. If a session doesn’t exist, the user is asked to enter their credentials. The IdP can choose how to authenticate the user - can be Username/Password, TOTP, MFA etc.</p><h3 id="IdP-sends-SAML-Assertion-to-SP"><a href="#IdP-sends-SAML-Assertion-to-SP" class="headerlink" title="IdP sends SAML Assertion to SP"></a>IdP sends SAML Assertion to SP</h3><p>Once the user is successfully authenticated, IdP sends back an XML message called “SAML Assertion” to the SP’s Assertion Consumer Service url. This contains the user’s details such as name, email, department etc and security measures (InResponseTo, IssueInstant). It’s also digitally signed so the SP can trust that the message is indeed from IdP and login the user into their system.</p><p><img src="/images/2020-saml/image-3.png" alt=""></p><h3 id="SP-creates-session-and-logs-in-user"><a href="#SP-creates-session-and-logs-in-user" class="headerlink" title="SP creates session and logs in user"></a>SP creates session and logs in user</h3><p>The user is now successfully logged in to the SP’s website! The SP will create a session for the user so the user can be automatically logged in the next time they visit the website.</p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>Hopefully this post is able to give you a high level overview of SAML authentication and how the SSO works.</p><p>Thanks for reading! :)</p><p>Related posts:<br><a href="https://www.sheshbabu.com/posts/implementing-saml-authentication-in-node-js/">Implementing SAML SSO in Node.js with Microsoft Entra ID</a></p>]]></content>
    
    <summary type="html">
    
      High level explanation of the SAML authentication protocol for beginners
    
    </summary>
    
    
      <category term="SAML" scheme="https://www.sheshbabu.com/tags/SAML/"/>
    
      <category term="Authentication" scheme="https://www.sheshbabu.com/tags/Authentication/"/>
    
  </entry>
  
  <entry>
    <title>Blocking usage of the any type in TypeScript codebases</title>
    <link href="https://www.sheshbabu.com/posts/blocking-usage-of-the-any-type-in-typescript-codebases/"/>
    <id>https://www.sheshbabu.com/posts/blocking-usage-of-the-any-type-in-typescript-codebases/</id>
    <published>2020-05-05T15:32:06.000Z</published>
    <updated>2024-05-19T06:17:28.270Z</updated>
    
    <content type="html"><![CDATA[<p>The <code>any</code> type in TypeScript denotes that the value can be anything - string, number, boolean etc. If widely used in a new codebase, it would defeat the purpose of using TypeScript over untyped JavaScript. So it’s preferable to block the usage of this type. If you’re incrementally migrating a JavaScript codebase to TypeScript, it would be better to use the new <a href="https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-0.html#new-unknown-top-type" target="_blank" rel="noopener">unknown</a> type.</p><h2 id="Blocking-implicit-any"><a href="#Blocking-implicit-any" class="headerlink" title="Blocking implicit any"></a>Blocking implicit any</h2><pre class=" language-javascript"><code class="language-javascript"><span class="token keyword">function</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token function">frobulate</span><span class="token punctuation">(</span><span class="token string">"hello"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">function</span> <span class="token function">frobulate</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span> <span class="token punctuation">{</span>  console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>The <code>name</code> argument in the above <code>frobulate</code> function has an implicit <code>any</code> type. Usually, TypeScript can infer types based on the control flow, assignments etc but in the above case, it’s hard to infer the type for the <code>name</code> argument so it would implicitly have the <code>any</code> type.</p><p>To block the implicit <code>any</code>, we can set <code>noImplicitAny</code> or <code>strict</code> to true in <code>tsconfig.json</code>.</p><h2 id="Blocking-explicit-any"><a href="#Blocking-explicit-any" class="headerlink" title="Blocking explicit any"></a>Blocking explicit any</h2><p>Setting <code>noImplicitAny</code> or <code>strict</code> to true would throw errors for the above code but if you explicitly type the <code>name</code> argument as <code>any</code>, it wouldn’t throw any errors.</p><pre class=" language-javascript"><code class="language-javascript"><span class="token keyword">function</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token function">frobulate</span><span class="token punctuation">(</span><span class="token string">"hello"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token comment" spellcheck="true">// Notice the "any" type added to the "name" argument</span><span class="token keyword">function</span> <span class="token function">frobulate</span><span class="token punctuation">(</span>name<span class="token punctuation">:</span> any<span class="token punctuation">)</span> <span class="token punctuation">{</span>  console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>To block explicit <code>any</code>, we need to use <a href="https://eslint.org" target="_blank" rel="noopener">ESLint</a> with the <code>typescript-eslint</code> plugin and turn on the <code>no-explicit-any</code> rule. Here’s an example <code>.eslintrc</code> config:</p><pre class=" language-json"><code class="language-json"><span class="token punctuation">{</span>  <span class="token property">"parser"</span><span class="token operator">:</span> <span class="token string">"@typescript-eslint/parser"</span><span class="token punctuation">,</span>  <span class="token property">"parserOptions"</span><span class="token operator">:</span> <span class="token punctuation">{</span>    <span class="token property">"sourceType"</span><span class="token operator">:</span> <span class="token string">"module"</span>  <span class="token punctuation">}</span><span class="token punctuation">,</span>  <span class="token property">"plugins"</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token string">"@typescript-eslint"</span><span class="token punctuation">]</span><span class="token punctuation">,</span>  <span class="token property">"rules"</span><span class="token operator">:</span> <span class="token punctuation">{</span>    <span class="token property">"@typescript-eslint/no-explicit-any"</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token string">"error"</span><span class="token punctuation">]</span>  <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre>]]></content>
    
    <summary type="html">
    
      Blocking usage of the any type in TypeScript codebases
    
    </summary>
    
    
      <category term="Development" scheme="https://www.sheshbabu.com/tags/Development/"/>
    
      <category term="TypeScript" scheme="https://www.sheshbabu.com/tags/TypeScript/"/>
    
      <category term="ESLint" scheme="https://www.sheshbabu.com/tags/ESLint/"/>
    
  </entry>
  
  <entry>
    <title>How to prevent code reviews from slowing down your team</title>
    <link href="https://www.sheshbabu.com/posts/how-to-prevent-code-reviews-from-slowing-down-your-team/"/>
    <id>https://www.sheshbabu.com/posts/how-to-prevent-code-reviews-from-slowing-down-your-team/</id>
    <published>2020-05-03T10:36:12.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<p>Code reviews when not implemented properly can seriously slow down your team’s ability to ship - with many changes stuck in review for multiple days (or weeks!) hurting your product’s time to market. Here are a few reasons why your code reviews might be taking a long time:</p><ul><li>Not having coding guidelines</li><li>Not using automated checks</li><li>Not doing self reviews</li><li>Raising huge PRs</li><li>Raising vague PRs</li><li>Not having deadlines for finishing reviews</li></ul><h2 id="Not-having-coding-guidelines"><a href="#Not-having-coding-guidelines" class="headerlink" title="Not having coding guidelines"></a>Not having coding guidelines</h2><p>Every team should have a set of coding guidelines which everyone in the team agrees to. This should contain things like naming conventions, folder structure, code formatting and best practices like unit testing, validations etc.</p><p>Not having clear guidelines and conventions would have every developer writing code the way they like, resulting in a lot of arguments during code review. If you notice a lot of comments about formatting, naming conventions etc, you need to start a discussion regarding coding guidelines.</p><p>Your team can either come up with your own set of guidelines or you can start with established guidelines from other companies. Here’s an example from <a href="http://google.github.io/styleguide/" target="_blank" rel="noopener">Google</a></p><h2 id="Not-using-automated-checks"><a href="#Not-using-automated-checks" class="headerlink" title="Not using automated checks"></a>Not using automated checks</h2><p>Once you have the coding guidelines documented, explore ways you can use tools to check for compliance. Almost all languages have formatters, linters etc that you can use in pre-commit hooks and CI/CD.</p><p>These tools would go through the code, check for coding guideline violations and notify the author about them. The author gets to fix these issues before raising the PR which greatly reduces the noise in the PR. The more checks your team offloads to automation the more time you get as a reviewer to focus on bigger issues like design flaws, implementation gaps etc</p><h2 id="Not-doing-self-reviews"><a href="#Not-doing-self-reviews" class="headerlink" title="Not doing self reviews"></a>Not doing self reviews</h2><p>Before asking others to review, the author needs to review their own changes. This is called “self review” and it’s similar to proofreading your email for typos and mistakes before sending it to others.</p><p>In practice, reviewing your own code is challenging as you’re mentally blind to its shortcomings. Here are some ways you can do better self reviews:</p><ul><li>Raise the PR without assigning any reviewers and come back to it after a couple of hours so you can see your changes with a fresh set of eyes</li><li>Fight the natural tendency to skim through your changes instead, go over your changes line-by-line deliberately</li><li>Follow a checklist for self reviews - “Is the code following the coding guidelines?”, “Have I covered all the business requirements by these changes?”, “Have I written tests for all the possible use cases?” etc</li></ul><h2 id="Raising-huge-PRs"><a href="#Raising-huge-PRs" class="headerlink" title="Raising huge PRs"></a>Raising huge PRs</h2><p>The number of review comments a PR gets is inversely proportional to the number of changes it contains. That is, large PRs =&gt; few comments, small PRs =&gt; many comments</p><blockquote><p>The number of review comments a PR gets is inversely proportional to the number of changes it contains</p></blockquote><p>This because reviewers get overwhelmed with large PRs and they might skim through the changes to wrap it up quickly. This defeats the purpose of code reviews. Sometimes, the opposite happens where a PR sits without any comments for many days because the reviewers are intimidated to start reviewing it.</p><p>Break down the PR into smaller chunks that make sense in isolation. You would get the reviewers to review it properly and promptly.</p><h2 id="Raising-vague-PRs"><a href="#Raising-vague-PRs" class="headerlink" title="Raising vague PRs"></a>Raising vague PRs</h2><p>It’s common to get asked to review PRs that have vague or no description about what it does. The reviewer then has to make sense of the changes by trying to remember the task from standups or issue tracker etc. Add details such as:</p><ul><li>What changes does this PR contain?</li><li>Which files should the reviewer start from?</li><li>Link to the task in the issue tracker</li><li>Screenshots if it’s a visual change</li></ul><p>Adding these would give more context to the reviewer and thereby helping them to review your PRs faster.</p><h2 id="Not-having-deadlines-for-finishing-reviews"><a href="#Not-having-deadlines-for-finishing-reviews" class="headerlink" title="Not having deadlines for finishing reviews"></a>Not having deadlines for finishing reviews</h2><p>One way for PRs to take forever to merge is to not have any deadlines for the reviewers to finish reviewing. Have reasonable deadlines such as:</p><ul><li>Reviews should be done within 48hrs from the time of raising the PR</li><li>Reviews for hotfixes need to be done within 30mins</li></ul><p>Thanks for reading! :)</p>]]></content>
    
    <summary type="html">
    
      How to prevent code reviews from slowing down your team
    
    </summary>
    
    
      <category term="Development" scheme="https://www.sheshbabu.com/tags/Development/"/>
    
      <category term="Code Reviews" scheme="https://www.sheshbabu.com/tags/Code-Reviews/"/>
    
  </entry>
  
  <entry>
    <title>Ways to manage config in frontend and their tradeoffs</title>
    <link href="https://www.sheshbabu.com/posts/frontend-config-management/"/>
    <id>https://www.sheshbabu.com/posts/frontend-config-management/</id>
    <published>2020-05-01T03:43:36.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<p>Let’s say our app has 3 environments - development, staging and production. We might have different configs for each of these environments like backend endpoints, analytics api keys, error reporting api keys, settings like turning off debug logs in production environment etc.</p><p><img src="/images/2020-config/image-1.png" alt=""></p><p>There are three common ways to manage these configs:</p><ol><li>Buildtime injection - For each environment, create a separate bundle and inject environment specific config into code.</li><li>Runtime resolution - Store the configs for all environments in the same bundle and dynamically resolve the right config in runtime.</li><li>Remote fetching - Fetch the config from backend during runtime</li></ol><p>Let’s talk about each of these approaches and their pros and cons.</p><h2 id="Buildtime-injection"><a href="#Buildtime-injection" class="headerlink" title="Buildtime injection"></a>Buildtime injection</h2><p><a href="https://en.wikipedia.org/wiki/Single-page_application" target="_blank" rel="noopener">SPA</a> apps are usually built using a bundler and deployed to web servers or copied into a nginx docker image and deployed to container platforms.</p><p>In this approach, a separate bundle (or docker image) is created for each environment and the config values for that environment is injected into the bundle using “environmental variables”. Of course, there’s no such thing as environment variables in frontend so it’s usually mimicked using global variables.</p><p>If you’re using <a href="https://webpack.js.org/plugins/define-plugin/" target="_blank" rel="noopener">webpack</a>, <a href="https://create-react-app.dev/docs/adding-custom-environment-variables/" target="_blank" rel="noopener">create-react-app</a> or <a href="https://cli.vuejs.org/guide/mode-and-env.html#environment-variables" target="_blank" rel="noopener">vue-cli</a> etc, you can access the config value as <code>process.env.YOUR_CONFIG_NAME</code> etc</p><p>Drawbacks:</p><ul><li>Since your environment specific config is baked into your bundle, you need to build separate bundles for each environment.</li><li>The way to access the config feels hacky - you need to get it from the <code>process.env</code> object which isn’t a part of the browser apis.</li><li>You cannot use this if you’re not using a bundler or a build tool.</li><li>Since the config values would be injected from CI/CD, the values would most likely not be version controlled.</li></ul><p>Advantages:</p><ul><li>Since each bundle contains only the config for a specific environment, attackers won’t be able to learn about the existence of other environments and their configs like backend endpoints etc by looking at the bundle. This provides a small safety in form of <a href="https://en.wikipedia.org/wiki/Security_through_obscurity" target="_blank" rel="noopener">security through obscurity</a> but shouldn’t be seriously relied upon.</li></ul><h2 id="Runtime-resolution"><a href="#Runtime-resolution" class="headerlink" title="Runtime resolution"></a>Runtime resolution</h2><p>Alternatively, we can store the configs for all environments in the source code and dynamically load the appropriate config for each environment. Usually, different environments would have different urls and we can use this to load the correct config.</p><p>Let’s say we have 3 environments - development, staging and production, which have the following urls - <code>dev.myapp.com</code>, <code>staging.myapp.com</code> and <code>www.myapp.com</code> respectively.</p><p>The configs for these 3 environments can be stored in <code>dev.config.json</code>, <code>staging.config.json</code> and <code>prod.config.json</code></p><pre class=" language-json"><code class="language-json">// dev.config.json<span class="token punctuation">{</span>  <span class="token property">"API_URL"</span><span class="token operator">:</span> <span class="token string">"dev.myapp.com/api"</span><span class="token punctuation">,</span>  <span class="token property">"ANALYTICS_KEY"</span><span class="token operator">:</span> <span class="token string">"xxxaaa"</span><span class="token punctuation">,</span>  <span class="token property">"ERROR_REPORTING_KEY"</span><span class="token operator">:</span> <span class="token string">"xxxeee"</span><span class="token punctuation">,</span>  <span class="token property">"IS_LOGGING_ENABLED"</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">}</span></code></pre><pre class=" language-json"><code class="language-json">// staging.config.json<span class="token punctuation">{</span>  <span class="token property">"API_URL"</span><span class="token operator">:</span> <span class="token string">"staging.myapp.com/api"</span><span class="token punctuation">,</span>  <span class="token property">"ANALYTICS_KEY"</span><span class="token operator">:</span> <span class="token string">"xxxaaa"</span><span class="token punctuation">,</span>  <span class="token property">"ERROR_REPORTING_KEY"</span><span class="token operator">:</span> <span class="token string">"xxxeee"</span><span class="token punctuation">,</span>  <span class="token property">"IS_LOGGING_ENABLED"</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">}</span></code></pre><pre class=" language-json"><code class="language-json">// prod.config.json<span class="token punctuation">{</span>  <span class="token property">"API_URL"</span><span class="token operator">:</span> <span class="token string">"www.myapp.com/api"</span><span class="token punctuation">,</span>  <span class="token property">"ANALYTICS_KEY"</span><span class="token operator">:</span> <span class="token string">"yyyaaa"</span><span class="token punctuation">,</span>  <span class="token property">"ERROR_REPORTING_KEY"</span><span class="token operator">:</span> <span class="token string">"yyyeee"</span><span class="token punctuation">,</span>  <span class="token property">"IS_LOGGING_ENABLED"</span><span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">}</span></code></pre><p>We can use <code>window.location.hostname</code> to decide which config to load in runtime:</p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// Config.js</span><span class="token keyword">import</span> prod <span class="token keyword">from</span> <span class="token string">"./prod.config.json"</span><span class="token punctuation">;</span><span class="token keyword">import</span> staging <span class="token keyword">from</span> <span class="token string">"./staging.config.json"</span><span class="token punctuation">;</span><span class="token keyword">import</span> dev <span class="token keyword">from</span> <span class="token string">"./dev.config.json"</span><span class="token punctuation">;</span><span class="token keyword">let</span> config <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token keyword">switch</span> <span class="token punctuation">(</span>window<span class="token punctuation">.</span>location<span class="token punctuation">.</span>hostname<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">case</span> <span class="token string">"www.myapp.com"</span><span class="token punctuation">:</span>    config <span class="token operator">=</span> prod<span class="token punctuation">;</span>    <span class="token keyword">break</span><span class="token punctuation">;</span>  <span class="token keyword">case</span> <span class="token string">"staging.myapp.com"</span><span class="token punctuation">:</span>    config <span class="token operator">=</span> staging<span class="token punctuation">;</span>    <span class="token keyword">break</span><span class="token punctuation">;</span>  <span class="token keyword">case</span> <span class="token string">"dev.myapp.com"</span><span class="token punctuation">:</span>    config <span class="token operator">=</span> dev<span class="token punctuation">;</span>    <span class="token keyword">break</span><span class="token punctuation">;</span>  <span class="token keyword">default</span><span class="token punctuation">:</span>    config <span class="token operator">=</span> dev<span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">export</span> <span class="token keyword">default</span> config<span class="token punctuation">;</span></code></pre><p>This can be consumed in the application as</p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// ApiClient.js</span><span class="token keyword">import</span> axios <span class="token keyword">from</span> <span class="token string">"axios"</span><span class="token punctuation">;</span><span class="token keyword">import</span> Config <span class="token keyword">from</span> <span class="token string">"../Config"</span><span class="token punctuation">;</span><span class="token keyword">const</span> ApiClient <span class="token operator">=</span> axios<span class="token punctuation">.</span><span class="token function">create</span><span class="token punctuation">(</span><span class="token punctuation">{</span> baseURL<span class="token punctuation">:</span> Config<span class="token punctuation">.</span>API_URL <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token keyword">export</span> <span class="token keyword">default</span> ApiClient<span class="token punctuation">;</span></code></pre><p>Usually some config values would be repeated for different environments. For example, you might decide to use the same analytics api key for all non-production environments. We can make the configs <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself" target="_blank" rel="noopener">DRY</a> by extracting these common config values to common.config.json and merging the configs in Config.js</p><pre class=" language-json"><code class="language-json">// common.config.json<span class="token punctuation">{</span>  <span class="token property">"ANALYTICS_KEY"</span><span class="token operator">:</span> <span class="token string">"xxxaaa"</span><span class="token punctuation">,</span>  <span class="token property">"ERROR_REPORTING_KEY"</span><span class="token operator">:</span> <span class="token string">"xxxeee"</span><span class="token punctuation">,</span>  <span class="token property">"IS_LOGGING_ENABLED"</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">}</span></code></pre><pre class=" language-json"><code class="language-json">// dev.config.json<span class="token punctuation">{</span>  <span class="token property">"API_URL"</span><span class="token operator">:</span> <span class="token string">"dev.myapp.com/api"</span><span class="token punctuation">}</span></code></pre><pre class=" language-json"><code class="language-json">// staging.config.json<span class="token punctuation">{</span>  <span class="token property">"API_URL"</span><span class="token operator">:</span> <span class="token string">"staging.myapp.com/api"</span><span class="token punctuation">}</span></code></pre><pre class=" language-json"><code class="language-json">// prod.config.json<span class="token punctuation">{</span>  <span class="token property">"API_URL"</span><span class="token operator">:</span> <span class="token string">"www.myapp.com/api"</span><span class="token punctuation">,</span>  <span class="token property">"ANALYTICS_KEY"</span><span class="token operator">:</span> <span class="token string">"yyyaaa"</span><span class="token punctuation">,</span>  <span class="token property">"ERROR_REPORTING_KEY"</span><span class="token operator">:</span> <span class="token string">"yyyeee"</span><span class="token punctuation">,</span>  <span class="token property">"IS_LOGGING_ENABLED"</span><span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">}</span></code></pre><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// Config.js</span><span class="token keyword">import</span> common <span class="token keyword">from</span> <span class="token string">"./common.config.json"</span><span class="token punctuation">;</span><span class="token keyword">import</span> prod <span class="token keyword">from</span> <span class="token string">"./prod.config.json"</span><span class="token punctuation">;</span><span class="token keyword">import</span> staging <span class="token keyword">from</span> <span class="token string">"./staging.config.json"</span><span class="token punctuation">;</span><span class="token keyword">import</span> dev <span class="token keyword">from</span> <span class="token string">"./dev.config.json"</span><span class="token punctuation">;</span><span class="token keyword">let</span> config <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token keyword">switch</span> <span class="token punctuation">(</span>window<span class="token punctuation">.</span>location<span class="token punctuation">.</span>hostname<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">case</span> <span class="token string">"www.myapp.com"</span><span class="token punctuation">:</span>    config <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token operator">...</span>common<span class="token punctuation">,</span> <span class="token operator">...</span>prod <span class="token punctuation">}</span><span class="token punctuation">;</span>    <span class="token keyword">break</span><span class="token punctuation">;</span>  <span class="token keyword">case</span> <span class="token string">"staging.myapp.com"</span><span class="token punctuation">:</span>    config <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token operator">...</span>common<span class="token punctuation">,</span> <span class="token operator">...</span>staging <span class="token punctuation">}</span><span class="token punctuation">;</span>    <span class="token keyword">break</span><span class="token punctuation">;</span>  <span class="token keyword">case</span> <span class="token string">"dev.myapp.com"</span><span class="token punctuation">:</span>    config <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token operator">...</span>common<span class="token punctuation">,</span> <span class="token operator">...</span>dev <span class="token punctuation">}</span><span class="token punctuation">;</span>    <span class="token keyword">break</span><span class="token punctuation">;</span>  <span class="token keyword">default</span><span class="token punctuation">:</span>    config <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token operator">...</span>common<span class="token punctuation">,</span> <span class="token operator">...</span>dev <span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">export</span> <span class="token keyword">default</span> config<span class="token punctuation">;</span></code></pre><p>Advantages</p><ul><li>Same bundle (or docker image) can be used for all environments</li><li>Works for projects that don’t use bundlers or build tools</li><li>The config values are version controlled and easy to reference as they’re in the same codebase</li></ul><p>Drawbacks</p><ul><li>You would be exposing the existence of other environments to attackers. Might be an issue if you’re not confident of the security of your undelying infra.</li></ul><h2 id="Remote-fetching"><a href="#Remote-fetching" class="headerlink" title="Remote fetching"></a>Remote fetching</h2><p>Generally, frontend builds don’t take much time so it’s easy to deploy config changes to users within minutes. But sometimes you might need to change the config from backend:</p><ul><li>Use a different set of configs for different user segments - A/B testing, phased rollouts, feature flags etc</li><li>Bypass processes - some enterprises have a lot of processes to go through to deploy a single change to a production site. It’s useful to fetch some configs from backend in those cases.</li></ul><p>Despite these advantages, the fact that we’re fetching these from backend complicates things - we need to design the code/interface with latency in mind and implement caching in a way that it doesn’t erode the advantages mentioned above. This approach is commonly used alongside either of the first 2 approaches.</p><p>Thanks for reading! :)</p>]]></content>
    
    <summary type="html">
    
      Ways to manage config in frontend and their tradeoffs
    
    </summary>
    
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
  </entry>
  
  <entry>
    <title>Minimal Viable Search using Postgres</title>
    <link href="https://www.sheshbabu.com/posts/minimal-viable-search-using-postgres/"/>
    <id>https://www.sheshbabu.com/posts/minimal-viable-search-using-postgres/</id>
    <published>2019-12-01T11:40:19.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<p>If you’re building a product, you might have deprioritized building the search feature thinking that it might take a long time to build. If you happen to be using Postgres, let me show you a quick and easy way to implement the search functionality.</p><h2 id="Test-drive"><a href="#Test-drive" class="headerlink" title="Test drive"></a>Test drive</h2><p>Let’s say you’re building an ecommerce app and you want to be able to search on the product descriptions. This can be done using the following query:</p><pre class=" language-sql"><code class="language-sql"><span class="token keyword">SELECT</span>  <span class="token operator">*</span><span class="token keyword">FROM</span>  products<span class="token keyword">WHERE</span>  to_tsvector<span class="token punctuation">(</span>description<span class="token punctuation">)</span> @@ websearch_to_tsquery<span class="token punctuation">(</span><span class="token string">'chocolate milk'</span><span class="token punctuation">)</span> <span class="token operator">=</span> <span class="token boolean">TRUE</span></code></pre><p>If you have a test database lying around, you can quickly try this out by replacing the table name, column name and search query. If you’re using Postgres 10 or below, “websearch_to_tsquery” won’t work. use “plainto_tsquery” instead.</p><p>Now, you might be having a lot of questions like:</p><ul><li>“to_tsvector”, “websearch_to_tsquery”, “@@” look weird!</li><li>How’s this different from “LIKE”?</li><li>How to make this faster?</li><li>What are the tradeoffs compared to ElasticSearch?</li></ul><h2 id="ts-what"><a href="#ts-what" class="headerlink" title="ts_what?"></a>ts_what?</h2><p>“ts” stands for Text Search.</p><p>At the very minimum, you need to only learn four things:</p><ul><li>Use “to_tsvector” function on the columns you’re searching on</li><li>Use “websearch_to_tsquery” function for the search query</li><li>Use the match operator “@@” to see if the above two match</li><li>Use “ts_rank” function to sort the results based on relevancy</li></ul><p>In simple terms, <code>to_tsvector</code> breaks down text into list of keywords and their positions.</p><p>Running:</p><pre class=" language-sql"><code class="language-sql"><span class="token keyword">SELECT</span> to_tsvector<span class="token punctuation">(</span><span class="token string">'A journey of a thousand miles begins with a single step'</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>gives</p><pre class=" language-sql"><code class="language-sql"><span class="token string">'begin'</span>:<span class="token number">7</span> <span class="token string">'journey'</span>:<span class="token number">2</span> <span class="token string">'mile'</span>:<span class="token number">6</span> <span class="token string">'singl'</span>:<span class="token number">10</span> <span class="token string">'step'</span>:<span class="token number">11</span> <span class="token string">'thousand'</span>:<span class="token number">5</span></code></pre><p>Notice that the <a href="https://en.wikipedia.org/wiki/Stop_words" target="_blank" rel="noopener">words</a> “A”, “of” and “with” are removed as they’re not useful in searching, the word “single” is normalized to its root form “singl” so it appears in more searches, the word “miles” is reduced to its singular form. This also takes care of normalizing the text to lowercase and removing special characters.</p><p>The function <code>websearch_to_tsquery</code> converts the user submitted search term into something that Postgres can understand. You can use Google style search queries like</p><pre class=" language-sql"><code class="language-sql">jaguar speed <span class="token operator">-</span>caripad <span class="token operator">OR</span> iphone<span class="token string">"chocolate chip"</span> recipe</code></pre><p>You can also try other query functions like “plainto_tsquery” or “phraseto_tsquery” which have their own way of parsing the search queries.</p><p>The <code>@@</code> operator matches the above search query with text from column. You can also use the <code>||</code> operator to concatenate multiple columns together and search on them.</p><p>The function ts_rank is used for sorting the search results by relevancy. The way it determines relevancy is by looking at how frequent the search terms appear, how close together they appear, in what position they appear etc.</p><p>By now you should have a good idea about how this is different from normal LIKE or pattern matching.</p><h2 id="Making-it-faster"><a href="#Making-it-faster" class="headerlink" title="Making it faster"></a>Making it faster</h2><p>Instead of building tsvectors everytime we query using to_tsvector, we can store it in a separate column when the record is created/updated. For this, we create the following trigger:</p><pre class=" language-sql"><code class="language-sql"><span class="token keyword">CREATE</span> <span class="token operator">OR</span> REPLACE <span class="token keyword">FUNCTION</span> fn_on_product_insert_store_tsv<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token keyword">RETURNS</span> <span class="token keyword">trigger</span> <span class="token keyword">AS</span>$$<span class="token keyword">BEGIN</span>  NEW<span class="token punctuation">.</span>tsv :<span class="token operator">=</span> to_tsvector<span class="token punctuation">(</span>NEW<span class="token punctuation">.</span>description<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">return</span> NEW<span class="token punctuation">;</span><span class="token keyword">END</span><span class="token punctuation">;</span>$$LANGUAGE <span class="token string">'plpgsql'</span><span class="token punctuation">;</span><span class="token keyword">CREATE</span> <span class="token keyword">TRIGGER</span> trg_on_product_insert_store_tsvBEFORE <span class="token keyword">INSERT</span> <span class="token operator">OR</span> <span class="token keyword">UPDATE</span> <span class="token keyword">ON</span> products<span class="token keyword">FOR EACH ROW</span><span class="token keyword">EXECUTE</span> <span class="token keyword">PROCEDURE</span> fn_on_product_insert_create_tsv<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>Let’s also add an index on this column to make the queries faster:</p><pre class=" language-sql"><code class="language-sql"><span class="token keyword">CREATE</span> <span class="token keyword">INDEX</span> tsv_idx <span class="token keyword">ON</span> products <span class="token keyword">USING</span> gin<span class="token punctuation">(</span>tsv<span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>This should greatly speed up your seach queries.</p><h2 id="Comparison-to-ElasticSearch"><a href="#Comparison-to-ElasticSearch" class="headerlink" title="Comparison to ElasticSearch"></a>Comparison to ElasticSearch</h2><p>ElasticSearch is synonymous with product search these days so you need to be aware of the tradeoffs:</p><p>When Postgres is better than ElasticSearch:</p><ul><li>One less dependency to manage or get approval for</li><li>Faster time to market - can see how your users are using search and decide if you need to use ElasticSearch for more sophisticated search features</li><li>There’s a single source of truth for the data - no need to keep multiple datastores in sync</li></ul><p>When ElasticSearch is better than Postgres:</p><ul><li>If your team already has expertise in ElasticSearch</li><li>Scale search queries seperately from normal database queries</li><li>You need support for <a href="https://stackoverflow.com/questions/5321595/what-is-faceted-search" target="_blank" rel="noopener">facets</a>. Here’s a <a href="https://roamanalytics.com/2019/04/16/faceted-search-with-postgres-using-tsvector/" target="_blank" rel="noopener">simple implementation</a> of facets in Postgres</li><li>More flexible and sophisticated search features</li></ul>]]></content>
    
    <summary type="html">
    
      Minimal Viable Search using Postgres
    
    </summary>
    
    
      <category term="Postgres" scheme="https://www.sheshbabu.com/tags/Postgres/"/>
    
      <category term="Search" scheme="https://www.sheshbabu.com/tags/Search/"/>
    
  </entry>
  
  <entry>
    <title>Career advice for new developers</title>
    <link href="https://www.sheshbabu.com/posts/career-advice-for-new-developers/"/>
    <id>https://www.sheshbabu.com/posts/career-advice-for-new-developers/</id>
    <published>2019-11-30T11:29:19.000Z</published>
    <updated>2024-05-19T06:17:28.270Z</updated>
    
    <content type="html"><![CDATA[<p>I’ve been meaning to write this post for a very long time - not only for others but also for what I wish I was told when I first started working as a developer. Hope you find this useful!</p><h2 id="Take-care-of-your-health"><a href="#Take-care-of-your-health" class="headerlink" title="Take care of your health"></a>Take care of your health</h2><p>Software development is not a physically demanding work like farming or factory work, but you are spending a lot of hours sitting and starting at a screen.</p><p>You might be healthy now but unfortunately, as you age, your vision gets blurry, your wrists or back start hurting etc.</p><p>This might not happen to everyone - There are plenty of developers who are in their 40s, 50s and above without these issues, but why take chances?</p><p>Take a break from work now and then. Earth will still be spinning after you return from break.</p><h2 id="Learn-the-basics"><a href="#Learn-the-basics" class="headerlink" title="Learn the basics"></a>Learn the basics</h2><p>There is a huge gap between what’s taught in universities and how we work in the industry. Unless you’re lucky, you don’t learn about how to use a debugger, source control, terminal, regex etc when you start your first job.</p><p>Make time to learn about these foundational concepts as you go about your job. It’ll be useful for your entire career.</p><h2 id="Tools-don’t-make-you-better"><a href="#Tools-don’t-make-you-better" class="headerlink" title="Tools don’t make you better"></a>Tools don’t make you better</h2><p>Using Vi/Emacs doesn’t make you an awesome developer. Similarly, using Sublime Text or Notepad++ doesn’t make you a bad developer. Same applies for languages, frameworks etc.</p><p>If you can complete your tasks and feel productive with the tools, continue using it. Don’t tie your self worth with the tools you use.</p><h2 id="Don’t-outsource-your-decisions"><a href="#Don’t-outsource-your-decisions" class="headerlink" title="Don’t outsource your decisions"></a>Don’t outsource your decisions</h2><p>Don’t blindly do something just because Google/Facebook/etc are doing it.</p><p>Each software project is different in terms of scope, business value, people involved, their skills, how long the project is going to be maintained etc.</p><p>Engineering is all about tradeoffs. Ask yourself: in which context a particular advice makes sense. This is the only way you’ll learn.</p><p>Examples:</p><ul><li>Don’t blindly use Kubernetes/Golang/React/Mongo for a static website which can be written in plain HTML and served by a single Nginx box.</li><li>If business urgently wants a throwaway website that will be only used for 2 weeks, don’t try to build it using TDD etc</li></ul><h2 id="Don’t-be-a-code-monkey"><a href="#Don’t-be-a-code-monkey" class="headerlink" title="Don’t be a code monkey"></a>Don’t be a code monkey</h2><p>Your job is not to write code but to solve business problems.</p><p>So, when you’re given some problems or tasks, try to understand what business problem you’re solving. Ask as many questions as needed for you to understand. It’s okay.</p><p>Once you know what you’re solving, there are good chances that you can come up with better solutions or solutions that can be implemented faster. Propose these solutions, most of times they’ll get accepted. Else you’ll be told why your solution won’t work in that scenario. In which case, you have learnt something and can make better suggestions in future.</p><h2 id="Don’t-stop-learning"><a href="#Don’t-stop-learning" class="headerlink" title="Don’t stop learning"></a>Don’t stop learning</h2><p>You’ll be learning a lot of new technologies and processes now that you’re new. But over years, you’ll get comfortable and stop learning.</p><p>Technology advances super fast. While it’s impossible to learn everything new that’s coming up, it’s better to be aware of the new things that are slowly becoming the industry standard and learn about it when you have free time.</p><p>Your peers, leads, managers are all in the same boat. See if you already have learning sessions in your company. If not, you can suggest starting one and most of the time, people will be supportive.</p><h2 id="Keep-notes"><a href="#Keep-notes" class="headerlink" title="Keep notes"></a>Keep notes</h2><p>Unless you have a really good memory, try to take notes about a particular process, technology, how to do x etc.</p><p>Most of the time, we end up doing something that we did 2-3 years back (vertically centre something in CSS, implement login flow, upload a file to S3 etc).</p><p>When you have notes, you can reach out to it and implement the solution quickly instead of googling about how to do it and scrolling through pages and pages of StacknOverflow and GitHub issues.</p><p>Even better, blog about it so that it also helps others 😊</p>]]></content>
    
    <summary type="html">
    
      Career advice for new developers
    
    </summary>
    
    
      <category term="Development" scheme="https://www.sheshbabu.com/tags/Development/"/>
    
  </entry>
  
  <entry>
    <title>Organizing HTTP requests using the API module pattern</title>
    <link href="https://www.sheshbabu.com/posts/organizing-http-requests-using-api-module-pattern/"/>
    <id>https://www.sheshbabu.com/posts/organizing-http-requests-using-api-module-pattern/</id>
    <published>2019-08-17T08:47:02.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<p>Let’s say you’re writing a frontend for an online store. You would have to make requests to get the shopping cart, add items to the cart, get product details, search for a product, list all products, etc.</p><p>If you’re directly calling <code>fetch</code> or <code>axios</code> in your code, they would look something like this.</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// HomePage.js</span><span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">getAllProducts</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">const</span> headers <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token string">"x-secret-header"</span><span class="token punctuation">:</span> <span class="token string">"ssshhh!"</span> <span class="token punctuation">}</span><span class="token punctuation">;</span>  <span class="token keyword">const</span> response <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">fetch</span><span class="token punctuation">(</span><span class="token string">"/api/products"</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> headers <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">const</span> body <span class="token operator">=</span> <span class="token keyword">await</span> response<span class="token punctuation">.</span><span class="token function">json</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">return</span> body<span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// CartPage.js</span><span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">addToCart</span><span class="token punctuation">(</span>itemId<span class="token punctuation">,</span> qty<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">const</span> headers <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token string">"x-secret-header"</span><span class="token punctuation">:</span> <span class="token string">"ssshhh!"</span> <span class="token punctuation">}</span><span class="token punctuation">;</span>  <span class="token keyword">const</span> payload <span class="token operator">=</span> JSON<span class="token punctuation">.</span><span class="token function">stringify</span><span class="token punctuation">(</span><span class="token punctuation">{</span> itemId<span class="token punctuation">,</span> qty <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">const</span> response <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">fetch</span><span class="token punctuation">(</span><span class="token string">"/api/cart"</span><span class="token punctuation">,</span> <span class="token punctuation">{</span>    method<span class="token punctuation">:</span> <span class="token string">"POST"</span><span class="token punctuation">,</span>    headers<span class="token punctuation">,</span>    body<span class="token punctuation">:</span> payload  <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">const</span> body <span class="token operator">=</span> <span class="token keyword">await</span> response<span class="token punctuation">.</span><span class="token function">json</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">return</span> body<span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p><img src="/images/2019-api/image-1.png" alt=""></p><p>This is totally fine if you’re only making a handful of requests, but if your codebase makes a lot of HTTP requests, it might be better to abstract them into their own modules instead of calling them directly.</p><h2 id="What-is-an-API-module"><a href="#What-is-an-API-module" class="headerlink" title="What is an API module?"></a>What is an API module?</h2><p>An API module is just a <a href="https://v8.dev/features/modules" target="_blank" rel="noopener">JS module</a> that contains HTTP logic organized by business domain. For the online store example, the business domains would be <code>Cart</code>, <code>Search</code>, <code>Inventory</code>, <code>Product Catalog</code>, <code>Order</code>, etc. The respective API modules would be <code>CartApi</code>, <code>SearchApi</code>, <code>InventoryApi</code>, <code>CatalogApi</code> and <code>OrderApi</code>.</p><p>Let’s rewrite the above code using the API module pattern by creating <code>CartApi</code> and <code>CatalogApi</code> modules.</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// CatalogApi.js</span><span class="token keyword">export</span> <span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">getAllProducts</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">const</span> headers <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token string">"x-secret-header"</span><span class="token punctuation">:</span> <span class="token string">"ssshhh!"</span> <span class="token punctuation">}</span><span class="token punctuation">;</span>  <span class="token keyword">const</span> response <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">fetch</span><span class="token punctuation">(</span><span class="token string">"/api/products"</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> headers <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">const</span> body <span class="token operator">=</span> <span class="token keyword">await</span> response<span class="token punctuation">.</span><span class="token function">json</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">return</span> body<span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// CartApi.js</span><span class="token keyword">export</span> <span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">addToCart</span><span class="token punctuation">(</span>itemId<span class="token punctuation">,</span> qty<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">const</span> headers <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token string">"x-secret-header"</span><span class="token punctuation">:</span> <span class="token string">"ssshhh!"</span> <span class="token punctuation">}</span><span class="token punctuation">;</span>  <span class="token keyword">const</span> payload <span class="token operator">=</span> JSON<span class="token punctuation">.</span><span class="token function">stringify</span><span class="token punctuation">(</span><span class="token punctuation">{</span> itemId<span class="token punctuation">,</span> qty <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">const</span> response <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">fetch</span><span class="token punctuation">(</span><span class="token string">"/api/cart"</span><span class="token punctuation">,</span> <span class="token punctuation">{</span>    method<span class="token punctuation">:</span> <span class="token string">"POST"</span><span class="token punctuation">,</span>    headers<span class="token punctuation">,</span>    body<span class="token punctuation">:</span> payload  <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">const</span> body <span class="token operator">=</span> <span class="token keyword">await</span> response<span class="token punctuation">.</span><span class="token function">json</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token keyword">return</span> body<span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>Now we can import these into the UI modules:</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// HomePage.js</span><span class="token keyword">import</span> <span class="token operator">*</span> <span class="token keyword">as</span> CatalogApi <span class="token keyword">from</span> <span class="token string">"../api/CatalogApi"</span><span class="token punctuation">;</span>CatalogApi<span class="token punctuation">.</span><span class="token function">getAllProducts</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// CartPage.js</span><span class="token keyword">import</span> <span class="token operator">*</span> <span class="token keyword">as</span> CartApi <span class="token keyword">from</span> <span class="token string">"../api/CartApi"</span><span class="token punctuation">;</span>CartApi<span class="token punctuation">.</span><span class="token function">addToCart</span><span class="token punctuation">(</span>itemId<span class="token punctuation">,</span> qty<span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p><img src="/images/2019-api/image-3.png" alt=""></p><h2 id="Benefits"><a href="#Benefits" class="headerlink" title="Benefits"></a>Benefits</h2><ul><li>Improve readability and testability by abstracting the HTTP code away from the UI or business code</li><li>One place to make modifications like renaming payload structure, query param names etc</li><li>One place to massage response into something that’s useful for the other parts of the codebase</li><li>Organizing HTTP code by business domain thereby improving code discoverability by new members of the team</li><li>Colocate custom app status codes sent by the server</li></ul><h2 id="HttpClient"><a href="#HttpClient" class="headerlink" title="HttpClient"></a>HttpClient</h2><p>If we see that there’s a lot of common code used in API modules, we can add one more layer called <code>HttpClient</code> or <code>ApiClient</code> to keep them <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself" target="_blank" rel="noopener">DRY</a>. The common code can be things like:</p><ul><li>Adding extra headers</li><li>Logging</li><li>Using right config for production, development etc — hostnames, headers, etc</li><li>Session handling etc</li></ul><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// HttpClient</span><span class="token keyword">export</span> <span class="token keyword">default</span> <span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">request</span><span class="token punctuation">(</span>path<span class="token punctuation">,</span> method<span class="token punctuation">,</span> body<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">const</span> headers <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// add custom headers here</span>  <span class="token keyword">const</span> url <span class="token operator">=</span> <span class="token string">""</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// add logic to get correct url for the environment</span>  <span class="token keyword">const</span> response <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">fetch</span><span class="token punctuation">(</span>url<span class="token punctuation">,</span> <span class="token punctuation">{</span> method<span class="token punctuation">,</span> body<span class="token punctuation">,</span> headers <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token comment" spellcheck="true">// add response handling</span>  <span class="token comment" spellcheck="true">// - session management</span>  <span class="token comment" spellcheck="true">// - convert to correct data type - response.json(), etc</span>  <span class="token comment" spellcheck="true">// - handle common errors like 401, 403 etc</span>  <span class="token keyword">return</span> response<span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p>We can now update our API modules to use HttpClient:</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// CatalogApi.js</span><span class="token keyword">import</span> <span class="token operator">*</span> <span class="token keyword">as</span> HttpClient <span class="token keyword">from</span> <span class="token string">"./HttpClient"</span><span class="token punctuation">;</span><span class="token keyword">export</span> <span class="token keyword">function</span> <span class="token function">getAllProducts</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">return</span> HttpClient<span class="token punctuation">.</span><span class="token function">request</span><span class="token punctuation">(</span><span class="token string">"/api/products"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// CartApi.js</span><span class="token keyword">import</span> <span class="token operator">*</span> <span class="token keyword">as</span> HttpClient <span class="token keyword">from</span> <span class="token string">"./HttpClient"</span><span class="token punctuation">;</span><span class="token keyword">export</span> <span class="token keyword">function</span> <span class="token function">addToCart</span><span class="token punctuation">(</span>itemId<span class="token punctuation">,</span> qty<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">return</span> HttpClient<span class="token punctuation">.</span><span class="token function">request</span><span class="token punctuation">(</span><span class="token string">"/api/cart"</span><span class="token punctuation">,</span> <span class="token string">"POST"</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> itemId<span class="token punctuation">,</span> qty <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><p><img src="/images/2019-api/image-2.png" alt=""></p><p>Thanks for reading! :)</p>]]></content>
    
    <summary type="html">
    
      Improve readability and testability by abstracting the HTTP code from the UI or business code
    
    </summary>
    
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
      <category term="Maintainability" scheme="https://www.sheshbabu.com/tags/Maintainability/"/>
    
  </entry>
  
  <entry>
    <title>How we use Storybook for documentation and code reviews</title>
    <link href="https://www.sheshbabu.com/posts/how-we-use-storybook-for-documentation-and-code-reviews/"/>
    <id>https://www.sheshbabu.com/posts/how-we-use-storybook-for-documentation-and-code-reviews/</id>
    <published>2018-11-14T04:03:54.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<h2 id="What-are-components"><a href="#What-are-components" class="headerlink" title="What are components?"></a>What are components?</h2><p>Thinking in terms of components has made frontend development a lot easier to reason about and codebases much more maintainable compared to traditional architectures like MVC etc.</p><p>Components are like lego blocks — they’re self-contained UI elements which can be combined with other components to create bigger UI elements. A screen is actually a big component composed of multiple smaller components. It’s components all the way down!</p><p>To illustrate the above point, let’s take the <em>ProductListItem</em> component. This component is composed of multiple smaller components like <em>ProductItemPills, ProductItemBadge, ProductItemLabel, ProductPrice, AtcButton</em> etc (Sorry, naming is hard 😉). This component is used by the ProductList component to display a list of products which is again used by screen-level components like <em>Cart, MyList, OrderDetail</em> etc.</p><p><img src="/images/2018-storybook/image-1.png" alt=""></p><p>The property of components to be self-contained makes decoupling them from the host frontend application trivial. They can be easily extracted to a component library repo and shared with all the different frontend applications inside an organisation or even open sourced to be used by others.</p><p><a href="https://storybook.js.org/" target="_blank" rel="noopener">Storybook</a> is a tool for organising such a component library. We also use it for other purposes like documenting component states and communicating changes in PRs.</p><p><img src="/images/2018-storybook/image-5.png" alt=""></p><h2 id="Documenting-states"><a href="#Documenting-states" class="headerlink" title="Documenting states"></a>Documenting states</h2><p>In UI design, component “states” are the different ways a component visually varies. A classic example is a Button component with states like <em>Default, Hover, Disabled, Loading</em> etc</p><p><img src="/images/2018-storybook/image-4.png" alt=""></p><p>Now, while documenting these Button states are useful, this is not a strong value proposition for introducing a new tool like Storybook to your dev/design workflow. Let’s take a more useful example — <em>ProductListItem</em> and <em>ProductGridItem</em>. These components are very congested visually since they pack a lot of information into a tiny space.</p><p><img src="/images/2018-storybook/image-3.png" alt=""></p><p>Depending on the product attributes, the above components can have any of the below states and they can also have states depending on the screen they’re used in (You can’t remove a product from your <em>OrderDetail</em> screen etc). Few states of <em>ProductListItem</em> component:</p><p><img src="/images/2018-storybook/image-2.png" alt=""></p><p>While designing UIs, the less space you have the more challenging it is to add more information or elements to it. The <em>ProductListItem</em> and <em>ProductGridItem</em> components are also frequently changing. Before making a change, you’d want to first see the different states of these components so you can better understand the constraints and edge cases. For this, you can create a static image with different component states in Sketch or Photoshop as documentation, but this not 100% what customers see — There might be some differences between design and actual implementation. Or you can go through the millions of products and different screens in your frontend to see the different states. The <em>ProductListItem</em> is also responsive, so you might need to do this exercise in different form factors!</p><p>With Storybook, the above becomes simple. You just create an entry for the component and its different states. It also plugins that let you see the component in different form factors.</p><p>This is also useful for components that are visually simple but just have so many states as edge cases.</p><h2 id="Communicating-changes-in-PRs"><a href="#Communicating-changes-in-PRs" class="headerlink" title="Communicating changes in PRs"></a>Communicating changes in PRs</h2><p>Storybook is a webapp that needs no api backend and can be built into a bunch of html/css/js files that can be statically hosted in S3, Github Pages, Netlify, Firebase etc. Everytime someone merges their changes to master branch, we build Storybook and deploy it. The master branch is what we use for documentation.</p><p>We internally use the <a href="http://blog.ramanshalupau.com/multifeature-staging-environments" target="_blank" rel="noopener">multi-feature staging environment</a> with great success at RedMart (Thanks Raman!). For example, if you’re working a feature to implement dark mode in the git branch named “feature/darkmode”, when you push this to origin, the CI scripts automatically create a url “darkmode.alpha.redmart.com” for you to share it with your team. This is incredibly useful and it’s hard to image developing without this. The way it works is simple yet genius — every git branch is deployed to a separate directory which gets resolved to by the webserver when accessed via <em>&lt;git-branch-name&gt;.alpha.redmart.com</em></p><p><img src="/images/2018-storybook/image-6.png" alt=""></p><p>We’ve extended this concept by having a Storybook for every git branch that’s accessible by an unique url. For every PR, the reviewers can go through the Storybook for that feature branch to understand the changes made. This has also made demoing the new/updated component states to the team easier.</p><p>This idea can be extended even further by running visual regression tests for the feature branch against master branch’s Storybook to validate the build. Component testing libraries like Enzyme let you assert the DOM changes or event handling based on different inputs passed to a component. But there’s no way (that I know of) to assert the visual changes happening to a component for different inputs. Visual regression testing for Storybooks could help fill that void. Storybook acts as test suites/specs and visual regression tests are the assertions.</p><p>Thanks for reading! :)</p><p>Originally published in <a href="http://geeks.redmart.com/2018/11/14/how-we-use-storybook-for-documentation-and-code-reviews/" target="_blank" rel="noopener">geeks.redmart.com</a></p>]]></content>
    
    <summary type="html">
    
      How we use Storybook for documentation and code reviews
    
    </summary>
    
    
      <category term="Development" scheme="https://www.sheshbabu.com/tags/Development/"/>
    
      <category term="React" scheme="https://www.sheshbabu.com/tags/React/"/>
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
      <category term="Code Reviews" scheme="https://www.sheshbabu.com/tags/Code-Reviews/"/>
    
      <category term="Maintainability" scheme="https://www.sheshbabu.com/tags/Maintainability/"/>
    
  </entry>
  
  <entry>
    <title>RedMart’s experience with PWA</title>
    <link href="https://www.sheshbabu.com/posts/redmart-experience-with-pwa/"/>
    <id>https://www.sheshbabu.com/posts/redmart-experience-with-pwa/</id>
    <published>2018-10-25T04:39:35.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<h2 id="What-is-a-PWA"><a href="#What-is-a-PWA" class="headerlink" title="What is a PWA?"></a>What is a PWA?</h2><p>Progressive Web App (PWA) is a term used for websites that behave like an app.</p><p>The phrase “behaves like an app” is different for different people. Some say an “app” is something that’s installed on your device, for some it’s the smooth and fluid user experience, for some it’s the ability to work offline, for others it’s the tight integration with OS like share menu, push notifications, widgets, 3d touch etc</p><p>Since the definition of an “app” is subjective, it’s better to have some baseline criteria for PWAs so we know everyone’s talking about the same thing. According to <a href="https://twitter.com/slightlylate" target="_blank" rel="noopener">Alex Russell</a>, one of the people who coined the term PWA, a website is a PWA if it satisfies the following <a href="https://infrequently.org/2016/09/what-exactly-makes-something-a-progressive-web-app/" target="_blank" rel="noopener">criteria</a>:</p><ul><li>Served over HTTPS</li><li>Loads offline even without internet connectivity</li><li>Contains a <a href="https://developer.mozilla.org/en-US/docs/Web/Manifest" target="_blank" rel="noopener">Web App Manifest</a></li></ul><h2 id="How-it-all-started"><a href="#How-it-all-started" class="headerlink" title="How it all started"></a>How it all started</h2><p>At RedMart, we embarked on our PWA journey 2 years back. We didn’t exactly set out to build a PWA since we had no idea what the benefits were. We started by adding <a href="https://developers.google.com/web/fundamentals/primers/service-workers/" target="_blank" rel="noopener">service workers</a> to cache some network requests in our mobile site. After this we realized that we actually satisfy 4 out of 5 <a href="https://developers.google.com/web/fundamentals/app-install-banners/#criteria" target="_blank" rel="noopener">criteria</a> for installable webapps. The only thing missing was a Web App Manifest. We quickly added that in 1-2 hours and bam, our mobile site is now a PWA!</p><h2 id="How-does-it-work"><a href="#How-does-it-work" class="headerlink" title="How does it work?"></a>How does it work?</h2><p><img src="/images/2018-pwa/image-1.png" alt=""></p><h2 id="What-were-the-benefits"><a href="#What-were-the-benefits" class="headerlink" title="What were the benefits?"></a>What were the benefits?</h2><p>When we first launched this to public, we didn’t know what to expect. Since the only extra cost was the development time to add the manifest, we weren’t overly concerned about how it performed. When we looked at the data, the results were unbelievable:</p><blockquote><p>The ecommerce conversion rate and the time spent on the installed PWA is almost as good as our native Android app and many times higher than the same mobile site used in browser!</p></blockquote><h2 id="How-can-we-make-this-even-better"><a href="#How-can-we-make-this-even-better" class="headerlink" title="How can we make this even better?"></a>How can we make this even better?</h2><p>We still have a lot of room for improvement. The current mobile site is a lightweight version of the RedMart app experience and we’re planning to add more features to improve the parity. This would increase the conversions and engagement. Adding push notifications for price drops, stock changes etc would help even more.</p><h2 id="Improving-the-install-experience"><a href="#Improving-the-install-experience" class="headerlink" title="Improving the install experience"></a>Improving the install experience</h2><p>The ability to install websites as apps is an interesting development in the last couple of years.</p><p>Browsers prompt users to install the website by showing a small <a href="https://developers.google.com/web/fundamentals/app-install-banners/#the_mini-info_bar" target="_blank" rel="noopener">infobar</a> or an <a href="https://developer.mozilla.org/en-US/docs/Web/Apps/Progressive/Add_to_home_screen#How_do_you_use_it" target="_blank" rel="noopener">icon in the address bar</a>. While this is much better than asking users to go to settings and select “Add to Home Screen”, this makes it harder for marketing to promote PWAs.</p><p>Native apps are usually promoted by having app install banners or popups in mobile site that when clicked on redirects the user to appstore to download the app.</p><p>It would be helpful to have apis that allows us to install the PWA. This seems to be a sensitive topic with a lot of <a href="https://github.com/w3c/manifest/issues/627" target="_blank" rel="noopener">back and forth</a> between developers and browser makers. Unfortunately, the app install popups look like they’re here to stay, so it might be useful to provide the apis so we can decide to install PWAs if the device supports it and if not redirect the user to app store.</p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>The future is pretty exciting for PWAs! As of this writing, Chrome has released support for <a href="https://developers.google.com/web/updates/2018/10/nic70#dpwa-windows" target="_blank" rel="noopener">desktop PWAs</a> and Microsoft allows <a href="https://docs.microsoft.com/en-us/microsoft-edge/progressive-web-apps/microsoft-store#submitting-your-pwa-manually" target="_blank" rel="noopener">submitting PWAs to Windows store</a></p><p>Each industry and userbase is different and you might see different results compared to us, but the barrier to entry is low and this can be quickly validated in a couple of days if you already have a mobile site. I hope this post was able to inspire you to adopt PWAs :)</p><p>Originally published in <a href="http://geeks.redmart.com/2018/10/25/redmarts-experience-with-pwa/" target="_blank" rel="noopener">geeks.redmart.com</a></p>]]></content>
    
    <summary type="html">
    
      RedMart’s experience with PWA
    
    </summary>
    
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
  </entry>
  
  <entry>
    <title>Packaging Node.js code into cross platform executables</title>
    <link href="https://www.sheshbabu.com/posts/packaging-node-js-code-into-cross-platform-executables/"/>
    <id>https://www.sheshbabu.com/posts/packaging-node-js-code-into-cross-platform-executables/</id>
    <published>2018-03-30T18:54:23.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<p>If you’re writing command line tools in Node.js, it can be hard to distribute them since users need to install Node.js in their machines before being able to use your tool. It would be a lot easier for users if we can package our app into a single executable file that they can download and run without installing anything extra.</p><p>We can use <a href="https://github.com/zeit/pkg" target="_blank" rel="noopener">pkg</a> to compile our code into a single executable file for multiple target platforms (Windows, Linux, Mac etc). </p><p>Let’s start with a simple example:</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">// index.js</span>console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"hello world"</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>This file prints “hello world” and exits. We can package this by running:</p><pre class=" language-shell"><code class="language-shell">$ npx pkg index.js</code></pre><p>This by default builds executables for three platforms - Windows, Linux and Mac:</p><pre class=" language-shell"><code class="language-shell">$ ls -1index-linuxindex-macosindex-win.exeindex.js</code></pre><p>The target platforms can be customized by using the <code>--targets</code> <a href="https://github.com/zeit/pkg#targets" target="_blank" rel="noopener">flag</a>.</p><p>I was curious how much space these take up:</p><pre class=" language-shell"><code class="language-shell">$ ls -lhtotal 183496-rwxr-xr-x  1 sheshbabu  staff    33M Mar 31 01:40 index-linux-rwxr-xr-x  1 sheshbabu  staff    34M Mar 31 01:40 index-macos-rw-r--r--  1 sheshbabu  staff    22M Mar 31 01:40 index-win.exe-rw-r--r--  1 sheshbabu  staff    28B Mar 31 00:46 index.js</code></pre><p>22-34MB feels like a bit too much for something that just prints “hello world”. Looking around the internet, it seems we can use tools like <a href="https://upx.github.io/" target="_blank" rel="noopener">upx</a> to reduce the file size.</p>]]></content>
    
    <summary type="html">
    
      Cross compile Node.js code into static binaries for easier distribution
    
    </summary>
    
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
      <category term="Node.js" scheme="https://www.sheshbabu.com/tags/Node-js/"/>
    
  </entry>
  
  <entry>
    <title>Tips for using ESLint in a legacy codebase</title>
    <link href="https://www.sheshbabu.com/posts/tips-for-using-eslint-in-a-legacy-codebase/"/>
    <id>https://www.sheshbabu.com/posts/tips-for-using-eslint-in-a-legacy-codebase/</id>
    <published>2018-02-24T18:28:07.000Z</published>
    <updated>2024-05-19T06:17:28.272Z</updated>
    
    <content type="html"><![CDATA[<p>ESLint is a fantastic tool that helps in detecting problematic patterns in codebase and enforcing coding conventions. It’s a must-have tool for any team trying to maintain high quality JavaScript codebases. </p><p>However, when you’re introducing ESLint to a legacy codebase for the first time, it’s hard not to feel overwhelmed by the bajillion errors it throws. </p><pre class=" language-shell"><code class="language-shell">$ eslint .✖ 20983 problems (20983 errors, 0 warnings)  19032 errors, 0 warnings potentially fixable with the `--fix` option.</code></pre><p>It’s understandable that the old code was written without the ESLint rules in mind but it becomes a problem when errors in any new code written gets drowned in all that noise. In this blog post, I’ll go through some techniques that can help you significantly reduce the amount of errors you see.</p><h2 id="Autofix"><a href="#Autofix" class="headerlink" title="Autofix"></a>Autofix</h2><p>ESLint has this very useful <a href="https://eslint.org/docs/user-guide/command-line-interface#--fix" target="_blank" rel="noopener">utility</a> that automatically fixes most of the errors. What can be and cannot be fixed depends on the rules themselves. The impact of this on reducing the number of errors depends on the codebase and the config used. If you notice in the above shell snippet, out of <code>20983</code> errors, <code>19032</code> can be automatically fixed! Let’s do that:</p><pre class=" language-shell"><code class="language-shell">$ eslint --fix .✖ 1700 problems (1700 errors, 0 warnings)</code></pre><p><code>1700</code> errors sounds a lot more manageable than <code>20983</code>!</p><h2 id="Specifying-environments-and-globals"><a href="#Specifying-environments-and-globals" class="headerlink" title="Specifying environments and globals"></a>Specifying environments and globals</h2><p>Each JS environment (Browser, Nodejs, etc) has it’s own set of host/global objects (window, process, setTimeout etc) in addition to native JS objects (Date, parseInt etc). ESLint doesn’t assume an environment, so you might see errors like <code>&#39;window&#39; is not defined</code> or <code>&#39;setTimeout&#39; is not defined</code>. By <a href="https://eslint.org/docs/user-guide/configuring#specifying-environments" target="_blank" rel="noopener">specifying environments</a>, these errors can be fixed.</p><p>Up until a few years back, the most common way of adding dependencies to a web codebase was to use <code>&lt;script&gt;</code> tags and they expose their apis using global variables like <code>$</code> for jQuery, <code>_</code> for underscore etc. You might see errors like <code>&#39;$&#39; is not defined</code> or <code>&#39;moment&#39; is not defined</code>. These errors can be fixed by <a href="https://eslint.org/docs/user-guide/configuring#specifying-globals" target="_blank" rel="noopener">specifying globals</a>.</p><h2 id="Disabling-rules"><a href="#Disabling-rules" class="headerlink" title="Disabling rules"></a>Disabling rules</h2><p>This should be the last resort and before we start disabling rules left and right, let’s go through the different levels in which we can disable rules:</p><h4 id="Line-level"><a href="#Line-level" class="headerlink" title="Line level"></a>Line level</h4><p>This disables rules for a line</p><pre class=" language-js"><code class="language-js"><span class="token function">alert</span><span class="token punctuation">(</span><span class="token string">'foo'</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment" spellcheck="true">// eslint-disable-line no-alert, quotes, semi</span><span class="token comment" spellcheck="true">// eslint-disable-next-line no-alert, quotes, semi</span><span class="token function">alert</span><span class="token punctuation">(</span><span class="token string">'foo'</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><h4 id="Block-level"><a href="#Block-level" class="headerlink" title="Block level"></a>Block level</h4><p>This disables rules for multiple lines</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">/* eslint-disable no-alert, no-console */</span><span class="token function">alert</span><span class="token punctuation">(</span><span class="token string">'foo'</span><span class="token punctuation">)</span><span class="token punctuation">;</span>console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">'bar'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token comment" spellcheck="true">/* eslint-enable no-alert, no-console */</span></code></pre><h4 id="File-level"><a href="#File-level" class="headerlink" title="File level"></a>File level</h4><p>This disables rules for the whole file</p><pre class=" language-js"><code class="language-js"><span class="token comment" spellcheck="true">/* eslint-disable no-alert */</span><span class="token function">alert</span><span class="token punctuation">(</span><span class="token string">'foo'</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><h4 id="Directory-level"><a href="#Directory-level" class="headerlink" title="Directory level"></a>Directory level</h4><p>This disables rules for a directory. This is done by creating a new config file inside that directory. There can be multiple ESLint config files inside a project. The closest config file would override the config files defined in outer directories.</p><p>This feature can be useful in situations where you need to disable some rules only a particular directory but still need those rules to be enabled for the rest of the codebase. For example, if you have <a href="https://eslint.org/docs/rules/no-magic-numbers" target="_blank" rel="noopener">no-magic-numbers</a> rule enabled and see a lot of errors from <code>tests/</code> directory because you’re using magic numbers in assertions, you can just disable this rule in a config file created inside the <code>tests/</code> directory.</p><p>Learn more about this feature <a href="https://eslint.org/docs/user-guide/configuring#configuration-cascading-and-hierarchy" target="_blank" rel="noopener">here</a>.</p><h4 id="Project-level"><a href="#Project-level" class="headerlink" title="Project level"></a>Project level</h4><p>You can disable rules for the whole codebase if you disable them in the root config file. Usually this approach is used when you’re using a <a href="https://eslint.org/docs/user-guide/configuring#using-a-shareable-configuration-package" target="_blank" rel="noopener">sharable config</a> like <a href="https://www.npmjs.com/package/eslint-config-airbnb" target="_blank" rel="noopener">eslint-config-airbnb</a> or <a href="https://www.npmjs.com/package/eslint-config-standard" target="_blank" rel="noopener">eslint-config-standard</a> etc and want to add your own overrides.</p><h4 id="Thoughts-on-disabling-rules"><a href="#Thoughts-on-disabling-rules" class="headerlink" title="Thoughts on disabling rules"></a>Thoughts on disabling rules</h4><p>It’s very easy to disable rules at the project level but doing so would mean that any new code written won’t be checked against those rules. </p><p>Disabling at line level is best as only those lines are affected but this can be very tedious as we might need to do this for thousands of lines.</p><p>Depending on the number of files that have errors, it’s better to disable at a file or directory level. This is easier than disabling at line level and the impact on new code is also smaller than disabling at a project level. </p><p>It would be nice to have a tool that goes through the codebase and adds the <code>eslint-disable-line</code> comment for the affected lines, but I don’t know if such tool exists. </p><h2 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h2><p>Using the steps above, all the errors in the codebase can be fixed or disabled. Any new errors would be only because of new code written and they can be fixed as soon as we see them. We can allocate some time depending on the bandwidth available to go through the disabled errors and fix them incrementally.</p><p>Thanks for reading! If you liked this, you might also like: <a href="http://www.sheshbabu.com/posts/speed-up-your-code-reviews-using-eslint-and-prettier/">Speed up your code reviews using ESLint and Prettier</a></p>]]></content>
    
    <summary type="html">
    
      Tips for using ESLint in a legacy codebase
    
    </summary>
    
    
      <category term="Development" scheme="https://www.sheshbabu.com/tags/Development/"/>
    
      <category term="ESLint" scheme="https://www.sheshbabu.com/tags/ESLint/"/>
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
  </entry>
  
  <entry>
    <title>Measuring response times of Express route handlers</title>
    <link href="https://www.sheshbabu.com/posts/measuring-response-times-of-express-route-handlers/"/>
    <id>https://www.sheshbabu.com/posts/measuring-response-times-of-express-route-handlers/</id>
    <published>2018-01-27T10:57:03.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<p>If we want to measure the total time taken for each route in an Express server, we can do so by adding a common middleware that tracks the time elapsed between the moment a request comes and before the response is sent back.</p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// index.js</span><span class="token keyword">const</span> express <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"express"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token keyword">const</span> logResponseTime <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"./response-time-logger"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token keyword">const</span> app <span class="token operator">=</span> <span class="token function">express</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>app<span class="token punctuation">.</span><span class="token function">use</span><span class="token punctuation">(</span>logResponseTime<span class="token punctuation">)</span><span class="token punctuation">;</span>app<span class="token punctuation">.</span><span class="token keyword">get</span><span class="token punctuation">(</span><span class="token string">"/"</span><span class="token punctuation">,</span> <span class="token punctuation">(</span>req<span class="token punctuation">,</span> res<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span>  res<span class="token punctuation">.</span><span class="token function">send</span><span class="token punctuation">(</span><span class="token string">"hello"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>app<span class="token punctuation">.</span><span class="token keyword">get</span><span class="token punctuation">(</span><span class="token string">"/slow"</span><span class="token punctuation">,</span> <span class="token punctuation">(</span>req<span class="token punctuation">,</span> res<span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span>  <span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">let</span> i <span class="token operator">=</span> <span class="token number">0</span><span class="token punctuation">;</span> i <span class="token operator">&lt;</span> <span class="token number">1e10</span><span class="token punctuation">;</span> i<span class="token operator">++</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><span class="token punctuation">}</span>  res<span class="token punctuation">.</span><span class="token function">send</span><span class="token punctuation">(</span><span class="token string">"hello"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>app<span class="token punctuation">.</span><span class="token function">listen</span><span class="token punctuation">(</span><span class="token number">3000</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// response-time-logger.js</span><span class="token keyword">function</span> <span class="token function">logResponseTime</span><span class="token punctuation">(</span>req<span class="token punctuation">,</span> res<span class="token punctuation">,</span> next<span class="token punctuation">)</span> <span class="token punctuation">{</span>  <span class="token keyword">const</span> startHrTime <span class="token operator">=</span> process<span class="token punctuation">.</span><span class="token function">hrtime</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  res<span class="token punctuation">.</span><span class="token function">on</span><span class="token punctuation">(</span><span class="token string">"finish"</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span>    <span class="token keyword">const</span> elapsedHrTime <span class="token operator">=</span> process<span class="token punctuation">.</span><span class="token function">hrtime</span><span class="token punctuation">(</span>startHrTime<span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token keyword">const</span> elapsedTimeInMs <span class="token operator">=</span> elapsedHrTime<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span> <span class="token operator">*</span> <span class="token number">1000</span> <span class="token operator">+</span> elapsedHrTime<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span> <span class="token operator">/</span> <span class="token number">1e6</span><span class="token punctuation">;</span>    console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"%s : %fms"</span><span class="token punctuation">,</span> req<span class="token punctuation">.</span>path<span class="token punctuation">,</span> elapsedTimeInMs<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span>module<span class="token punctuation">.</span>exports <span class="token operator">=</span> logResponseTime<span class="token punctuation">;</span></code></pre><p>Calling the above two endpoints would log</p><pre><code>/ : 1.791791ms/slow : 18541.045675ms</code></pre><p>The <code>index.js</code> is the entrypoint to our server and that’s usually the place where all the common middlewares are added. The <code>logResponseTime</code> middleware needs to be at the top because we want to be as accurate as possible when we measure the time taken for each route in our Express server. The <a href="https://nodejs.org/api/process.html#process_process_hrtime_time" target="_blank" rel="noopener">process.hrtime</a> api is used to measure the elapsed time as it is more accurate than just using the <code>Date</code> api.</p>]]></content>
    
    <summary type="html">
    
      Measuring response times of Express route handlers
    
    </summary>
    
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
      <category term="Node.js" scheme="https://www.sheshbabu.com/tags/Node-js/"/>
    
      <category term="Express.js" scheme="https://www.sheshbabu.com/tags/Express-js/"/>
    
  </entry>
  
  <entry>
    <title>Speed up your code reviews using ESLint and Prettier</title>
    <link href="https://www.sheshbabu.com/posts/speed-up-your-code-reviews-using-eslint-and-prettier/"/>
    <id>https://www.sheshbabu.com/posts/speed-up-your-code-reviews-using-eslint-and-prettier/</id>
    <published>2017-09-07T12:00:00.000Z</published>
    <updated>2024-05-19T06:17:28.272Z</updated>
    
    <content type="html"><![CDATA[<p>Code reviews are very important if you want to build great software. It’s an effective way of sharing knowledge of the codebase to other members of the team, it’s a good opportunity to learn as the reviewers might suggest better ways of solving a problem than your usual approach, it helps in identifying logical bugs or gaps in implementation, it helps in ensuring that the codebase stays readable, maintainable and follows your team’s coding conventions etc</p><p>Code reviews are also time-consuming. For reviewers, it requires them to go through the changes to look for issues and opportunities for improvement. The more things they’re checking for, the more time consuming and less focussed they are. For the authors, once the review is over, they need to refactor the code as per the review comments, do additional testing, do self-review etc. Rinse and repeat.</p><p>We should strive to make this process faster so we can deliver the software to our users as quickly as possible. We can lessen the time it takes for reviewers to review the code by automating the code review as much as possible and letting them focus on the non-automatable aspects. For the authors, we can give feedback on the code early so they can refactor much earlier in the development process.</p><p>Checking whether the changes follow the coding conventions, best practices and code formatting is something we can automate. These are also the ones that trigger the most nitpicks during a code review and thereby generating more noise in the review comments.</p><p><a href="https://eslint.org/" target="_blank" rel="noopener">ESLint</a> and <a href="https://github.com/prettier/prettier" target="_blank" rel="noopener">Prettier</a> are two popular tools that can help us achieve this. Prettier <a href="https://github.com/prettier/prettier#what-does-prettier-do" target="_blank" rel="noopener">formats</a> code and ESLint helps enforce coding conventions and <a href="https://eslint.org/docs/rules/#best-practices" target="_blank" rel="noopener">find problematic patterns</a> in code. ESLint also has an <a href="https://eslint.org/docs/user-guide/command-line-interface#--fix" target="_blank" rel="noopener">auto-fix</a> mode that automatically fixes some of the rule violations. Both <a href="https://github.com/prettier/prettier#editor-integration" target="_blank" rel="noopener">have</a> <a href="https://eslint.org/docs/user-guide/integrations#editors" target="_blank" rel="noopener">plugins</a> for all popular editors which ensures that the violations are quickly shown to the developer. But if the developer is using an editor that doesn’t have these plugins or is someone who sporadically contributes code and you don’t want to add friction to their workflow by asking them to install or configure the plugins, we can use the git commit hooks so that the code gets automatically checked as it is committed. Git commit hooks are also useful in making sure that all the committed code adheres to the rules and there are no <a href="https://blog.codinghorror.com/the-broken-window-theory/" target="_blank" rel="noopener">broken windows</a> due to misconfigured editors or other reasons. You can use <a href="https://github.com/okonet/lint-staged" target="_blank" rel="noopener">lint-staged</a> for easily setting up git commit hooks. </p><p>If you’re newly setting up a project and don’t want to spend time initially to pick the rules or config, Prettier comes with good defaults and ESLint can be initialized with popular style guides.</p><p>If you want to introduce this to an existing project, you can run all the files through Prettier and use ESLint auto-fix to change the existing code as per the new rules. For the rules that are not covered by auto-fix, you can <a href="https://eslint.org/docs/user-guide/configuring#configuring-rules" target="_blank" rel="noopener">disable</a> all the remaining non-auto-fixable rules initially and fix them manually in batches and re-enable them as they’re fixed. If it’s a very larger project, you might want to split your codebase into different sections and have <a href="https://eslint.org/docs/user-guide/configuring#configuration-cascading-and-hierarchy" target="_blank" rel="noopener">directory specific ESLint configs</a> and make changes on one section at a time.</p>]]></content>
    
    <summary type="html">
    
      Speed up your code reviews using ESLint and Prettier
    
    </summary>
    
    
      <category term="Development" scheme="https://www.sheshbabu.com/tags/Development/"/>
    
      <category term="ESLint" scheme="https://www.sheshbabu.com/tags/ESLint/"/>
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
      <category term="Code Reviews" scheme="https://www.sheshbabu.com/tags/Code-Reviews/"/>
    
      <category term="Prettier" scheme="https://www.sheshbabu.com/tags/Prettier/"/>
    
  </entry>
  
  <entry>
    <title>Disabling Bunyan in tests</title>
    <link href="https://www.sheshbabu.com/posts/disabling-bunyan-in-tests/"/>
    <id>https://www.sheshbabu.com/posts/disabling-bunyan-in-tests/</id>
    <published>2017-07-25T14:03:33.000Z</published>
    <updated>2024-05-19T06:17:28.270Z</updated>
    
    <content type="html"><![CDATA[<p>Bunyan (as of v1.8.10) doesn’t provide an explicit api to mute logging. This is especially useful when you’re running unit tests and don’t want logs and test reports to be mixed. </p><p>One workaround <a href="https://github.com/trentm/node-bunyan/issues/456#issuecomment-259052991" target="_blank" rel="noopener">suggested</a> by the author is to set the log level to a value above <code>FATAL</code>.</p><p>Set an environment variable when running tests:</p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// package.json</span><span class="token operator">...</span><span class="token operator">...</span>  <span class="token string">"scripts"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span>    <span class="token string">"test"</span><span class="token punctuation">:</span> <span class="token string">"NODE_ENV=test mocha"</span>  <span class="token punctuation">}</span><span class="token operator">...</span><span class="token operator">...</span></code></pre><p>Check for it in the logger module and if it matches, set the log level above <code>FATAL</code>.</p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// logger.js</span><span class="token keyword">const</span> bunyan <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"bunyan"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token keyword">const</span> logger <span class="token operator">=</span> bunyan<span class="token punctuation">.</span><span class="token function">createLogger</span><span class="token punctuation">(</span><span class="token punctuation">{</span>name<span class="token punctuation">:</span> <span class="token string">"myapp"</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token keyword">if</span> <span class="token punctuation">(</span>process<span class="token punctuation">.</span>env<span class="token punctuation">.</span>NODE_ENV <span class="token operator">===</span> <span class="token string">"test"</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>  logger<span class="token punctuation">.</span><span class="token function">level</span><span class="token punctuation">(</span>bunyan<span class="token punctuation">.</span>FATAL <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span>module<span class="token punctuation">.</span>exports <span class="token operator">=</span> logger<span class="token punctuation">;</span></code></pre><p> This disables logging when running tests.</p>]]></content>
    
    <summary type="html">
    
      Disable Bunyan logs when running tests
    
    </summary>
    
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
      <category term="Testing" scheme="https://www.sheshbabu.com/tags/Testing/"/>
    
      <category term="Node.js" scheme="https://www.sheshbabu.com/tags/Node-js/"/>
    
      <category term="Bunyan" scheme="https://www.sheshbabu.com/tags/Bunyan/"/>
    
      <category term="Logging" scheme="https://www.sheshbabu.com/tags/Logging/"/>
    
  </entry>
  
  <entry>
    <title>Unit testing Express route handlers</title>
    <link href="https://www.sheshbabu.com/posts/unit-testing-express-route-handlers/"/>
    <id>https://www.sheshbabu.com/posts/unit-testing-express-route-handlers/</id>
    <published>2017-07-08T02:22:49.000Z</published>
    <updated>2024-05-19T06:17:28.272Z</updated>
    
    <content type="html"><![CDATA[<p>We can unit test <a href="https://expressjs.com/" target="_blank" rel="noopener">Express.js</a> route handler functions using a mocking library called <a href="https://github.com/howardabrams/node-mocks-http" target="_blank" rel="noopener">node-mocks-http</a></p><p>Let’s say we’ve a simple express app</p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// index.js</span><span class="token keyword">const</span> express <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"express"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token keyword">const</span> exampleRouter <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"./example-router"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token keyword">const</span> app <span class="token operator">=</span> <span class="token function">express</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>app<span class="token punctuation">.</span><span class="token function">use</span><span class="token punctuation">(</span><span class="token string">"/example"</span><span class="token punctuation">,</span> exampleRouter<span class="token punctuation">)</span><span class="token punctuation">;</span>app<span class="token punctuation">.</span><span class="token function">listen</span><span class="token punctuation">(</span><span class="token number">3000</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>With route handler defined separately as</p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// example-router.js</span><span class="token keyword">function</span> <span class="token function">exampleRouteHandler</span><span class="token punctuation">(</span>req<span class="token punctuation">,</span> res<span class="token punctuation">)</span> <span class="token punctuation">{</span>  res<span class="token punctuation">.</span><span class="token function">send</span><span class="token punctuation">(</span><span class="token string">"hello world!"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span>module<span class="token punctuation">.</span>exports <span class="token operator">=</span> exampleRouteHandler<span class="token punctuation">;</span></code></pre><p>For unit testing, we should be able to pass various inputs and see if we get correct outputs. Here, we should be able to pass valid request (<code>req</code>) and response (<code>res</code>) objects as inputs and since this function doesn’t return anything, we should be able to make assertions on the response object (<code>res</code>).</p><p>We can do this by using node-mocks-http’s <a href="https://github.com/howardabrams/node-mocks-http#createrequest" target="_blank" rel="noopener">createRequest</a> and <a href="https://github.com/howardabrams/node-mocks-http#createresponse" target="_blank" rel="noopener">createResponse</a> apis.</p><p>Let’s write a simple test for this using <a href="https://github.com/mochajs/mocha" target="_blank" rel="noopener">mocha</a></p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// example-router.test.js</span><span class="token keyword">const</span> assert <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"assert"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token keyword">const</span> httpMocks <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"node-mocks-http"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token keyword">const</span> exampleRouteHandler <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"./example-router"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token function">describe</span><span class="token punctuation">(</span><span class="token string">"Example Router"</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span>  <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">"should return 'hello world' for GET /example"</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span>    <span class="token keyword">const</span> mockRequest <span class="token operator">=</span> httpMocks<span class="token punctuation">.</span><span class="token function">createRequest</span><span class="token punctuation">(</span><span class="token punctuation">{</span>      method<span class="token punctuation">:</span> <span class="token string">"GET"</span><span class="token punctuation">,</span>      url<span class="token punctuation">:</span> <span class="token string">"/example"</span>    <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token keyword">const</span> mockResponse <span class="token operator">=</span> httpMocks<span class="token punctuation">.</span><span class="token function">createResponse</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token function">exampleRouteHandler</span><span class="token punctuation">(</span>mockRequest<span class="token punctuation">,</span> mockResponse<span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token keyword">const</span> actualResponseBody <span class="token operator">=</span> mockResponse<span class="token punctuation">.</span><span class="token function">_getData</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token keyword">const</span> expectedResponseBody <span class="token operator">=</span> <span class="token string">"hello world!"</span><span class="token punctuation">;</span>    <span class="token function">assert</span><span class="token punctuation">(</span>actualResponseBody<span class="token punctuation">,</span> expectedResponseBody<span class="token punctuation">)</span><span class="token punctuation">;</span>  <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>Checkout the <a href="https://github.com/howardabrams/node-mocks-http" target="_blank" rel="noopener">node-mocks-http repo</a> for more info.</p>]]></content>
    
    <summary type="html">
    
      Using node-mocks-http to write unit tests for Express route handlers
    
    </summary>
    
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
      <category term="Testing" scheme="https://www.sheshbabu.com/tags/Testing/"/>
    
      <category term="Node.js" scheme="https://www.sheshbabu.com/tags/Node-js/"/>
    
      <category term="Express.js" scheme="https://www.sheshbabu.com/tags/Express-js/"/>
    
  </entry>
  
  <entry>
    <title>Working with Fetch api</title>
    <link href="https://www.sheshbabu.com/posts/working-with-fetch-api/"/>
    <id>https://www.sheshbabu.com/posts/working-with-fetch-api/</id>
    <published>2016-09-25T10:20:00.000Z</published>
    <updated>2024-05-19T06:17:28.273Z</updated>
    
    <content type="html"><![CDATA[<p>Fetch is a much needed improvement over XHR, it simplifies making network requests by exposing an easy to use api and having promise support out of the box.</p><pre class=" language-javascript"><code class="language-javascript"><span class="token function">fetch</span><span class="token punctuation">(</span>url<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token keyword">function</span> <span class="token punctuation">(</span>response<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">return</span> response<span class="token punctuation">.</span><span class="token function">json</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><h3 id="Wrapping-fetch"><a href="#Wrapping-fetch" class="headerlink" title="Wrapping fetch"></a>Wrapping fetch</h3><p>While the above example is good enough for most cases, sometimes you might need to send the same headers in all the requests or handle all the  responses in the same way. Doing so in each and every <code>fetch</code> call would be duplicating a lot of code. This can solved by creating a wrapper around the fetch method and using that wrapper throughout the application instead of fetch.</p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// fetch-wrapper.js</span><span class="token keyword">function</span> <span class="token function">fetchWrapper</span><span class="token punctuation">(</span>url<span class="token punctuation">,</span> options<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">var</span> options <span class="token operator">=</span> options <span class="token operator">||</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">;</span>    options<span class="token punctuation">.</span>headers<span class="token punctuation">[</span><span class="token string">'Custom-Header'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token string">'Your custom header value here'</span><span class="token punctuation">;</span>    <span class="token keyword">return</span> <span class="token function">fetch</span><span class="token punctuation">(</span>url<span class="token punctuation">,</span> options<span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// books.js</span><span class="token function">fetchWrapper</span><span class="token punctuation">(</span><span class="token string">'/api/books'</span><span class="token punctuation">)</span>    <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token keyword">function</span> <span class="token punctuation">(</span>data<span class="token punctuation">)</span> <span class="token punctuation">{</span>        console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>data<span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><h3 id="Rejecting-on-HTTP-errors"><a href="#Rejecting-on-HTTP-errors" class="headerlink" title="Rejecting on HTTP errors"></a>Rejecting on HTTP errors</h3><p>Coming from jQuery.ajax, one of the main gotcha’s about fetch is that it does not reject on HTTP errors - It only rejects on network failures. While this makes sense because any response (whether 2xx or 4xx etc) is still a response and thereby a ‘success’, you might want fetch to reject on http errors so that the <code>catch</code> part of your promise chain can handle them appropriately.</p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// fetch-wrapper.js</span><span class="token keyword">function</span> <span class="token function">fetchWrapper</span><span class="token punctuation">(</span>url<span class="token punctuation">,</span> options<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">return</span> <span class="token function">fetch</span><span class="token punctuation">(</span>url<span class="token punctuation">,</span> options<span class="token punctuation">)</span>        <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span>handleResponse<span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">function</span> handleResponse <span class="token punctuation">(</span>response<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">if</span> <span class="token punctuation">(</span>response<span class="token punctuation">.</span>ok<span class="token punctuation">)</span> <span class="token punctuation">{</span>        <span class="token keyword">return</span> response<span class="token punctuation">.</span><span class="token function">json</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>        <span class="token keyword">throw</span> <span class="token keyword">new</span> <span class="token class-name">Error</span><span class="token punctuation">(</span>response<span class="token punctuation">.</span>statusText<span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// books.js</span><span class="token function">fetchWrapper</span><span class="token punctuation">(</span><span class="token string">'/api/books'</span><span class="token punctuation">)</span>    <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token keyword">function</span> <span class="token punctuation">(</span>data<span class="token punctuation">)</span> <span class="token punctuation">{</span>        console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>data<span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token punctuation">}</span><span class="token punctuation">)</span>    <span class="token punctuation">.</span><span class="token keyword">catch</span><span class="token punctuation">(</span><span class="token keyword">function</span> <span class="token punctuation">(</span>error<span class="token punctuation">)</span> <span class="token punctuation">{</span>        console<span class="token punctuation">.</span><span class="token function">error</span><span class="token punctuation">(</span>error<span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><h3 id="Handling-JSON-responses"><a href="#Handling-JSON-responses" class="headerlink" title="Handling JSON responses"></a>Handling JSON responses</h3><p>If all the responses are guaranteed to be JSON, then we can parse them before passing them down the promise chain. Since fetch throws TypeError on network errors, we can handle it in <code>handleNetworkError</code> to throw a JSON object similar to ones we get from our backend.</p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// fetch-wrapper.js</span><span class="token keyword">function</span> <span class="token function">fetchWrapper</span><span class="token punctuation">(</span>url<span class="token punctuation">,</span> options<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">return</span> <span class="token function">fetch</span><span class="token punctuation">(</span>url<span class="token punctuation">,</span> options<span class="token punctuation">)</span>        <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span>handleResponse<span class="token punctuation">,</span> handleNetworkError<span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token keyword">function</span> handleResponse <span class="token punctuation">(</span>response<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">if</span> <span class="token punctuation">(</span>response<span class="token punctuation">.</span>ok<span class="token punctuation">)</span> <span class="token punctuation">{</span>        <span class="token keyword">return</span> response<span class="token punctuation">.</span><span class="token function">json</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>        <span class="token keyword">return</span> response<span class="token punctuation">.</span><span class="token function">json</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token keyword">function</span> <span class="token punctuation">(</span>error<span class="token punctuation">)</span> <span class="token punctuation">{</span>            <span class="token keyword">throw</span> error<span class="token punctuation">;</span>        <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token keyword">function</span> handleNetworkError <span class="token punctuation">(</span>error<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">throw</span> <span class="token punctuation">{</span>        msg<span class="token punctuation">:</span> error<span class="token punctuation">.</span>message    <span class="token punctuation">}</span><span class="token punctuation">;</span><span class="token punctuation">}</span></code></pre><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// books.js</span><span class="token function">fetchWrapper</span><span class="token punctuation">(</span><span class="token string">'/api/books'</span><span class="token punctuation">)</span>    <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token keyword">function</span> <span class="token punctuation">(</span>data<span class="token punctuation">)</span> <span class="token punctuation">{</span>        console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>data<span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token punctuation">}</span><span class="token punctuation">)</span>    <span class="token punctuation">.</span><span class="token keyword">catch</span><span class="token punctuation">(</span><span class="token keyword">function</span> <span class="token punctuation">(</span>error<span class="token punctuation">)</span> <span class="token punctuation">{</span>        console<span class="token punctuation">.</span><span class="token function">error</span><span class="token punctuation">(</span>error<span class="token punctuation">.</span>msg<span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><h3 id="Timeouts"><a href="#Timeouts" class="headerlink" title="Timeouts"></a>Timeouts</h3><p>There’s no support for timeouts in the fetch api, though this can be achieved by creating a promise that rejects on timeout and using it with the Promise.race api.</p><pre class=" language-javascript"><code class="language-javascript"><span class="token keyword">function</span> timeout <span class="token punctuation">(</span>value<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">return</span> <span class="token keyword">new</span> <span class="token class-name">Promise</span><span class="token punctuation">(</span><span class="token keyword">function</span> <span class="token punctuation">(</span>resolve<span class="token punctuation">,</span> reject<span class="token punctuation">)</span> <span class="token punctuation">{</span>        <span class="token function">setTimeout</span><span class="token punctuation">(</span><span class="token keyword">function</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>            <span class="token function">reject</span><span class="token punctuation">(</span><span class="token keyword">new</span> <span class="token class-name">Error</span><span class="token punctuation">(</span><span class="token string">'Sorry, request timed out.'</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>        <span class="token punctuation">}</span><span class="token punctuation">,</span> value<span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">}</span>Promise<span class="token punctuation">.</span><span class="token function">race</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token function">timeout</span><span class="token punctuation">(</span><span class="token number">1000</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token function">fetch</span><span class="token punctuation">(</span>url<span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span>    <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token keyword">function</span> <span class="token punctuation">(</span>response<span class="token punctuation">)</span> <span class="token punctuation">{</span>        console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>response<span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token punctuation">}</span><span class="token punctuation">)</span>    <span class="token punctuation">.</span><span class="token keyword">catch</span><span class="token punctuation">(</span><span class="token keyword">function</span> <span class="token punctuation">(</span>error<span class="token punctuation">)</span> <span class="token punctuation">{</span>        console<span class="token punctuation">.</span><span class="token function">error</span><span class="token punctuation">(</span>error<span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre><p>But keep in mind that since fetch has no support for aborting the request, the above example only rejects the promise but the request itself is still alive. This behavior is different from XHR based libraries which abort the request when it takes longer than the timeout value.</p><h3 id="Browser-support"><a href="#Browser-support" class="headerlink" title="Browser support"></a>Browser support</h3><p>Fetch is <a href="http://caniuse.com/#feat=fetch" target="_blank" rel="noopener">supported in most browsers</a> and has a <a href="https://github.com/github/fetch" target="_blank" rel="noopener">polyfill</a> for those who don’t support it.</p><h3 id="Limitations"><a href="#Limitations" class="headerlink" title="Limitations"></a>Limitations</h3><p>Fetch is being developed iteratively and there are certain things that it does not support like <a href="https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Using_XMLHttpRequest#Monitoring_progress" target="_blank" rel="noopener">monitoring progress</a>, <a href="https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/abort" target="_blank" rel="noopener">aborting</a> a request etc. If these are absolutely necessary to your application, then your should use XHR or its abstractions like jQuery.ajax, axios etc instead of fetch.</p><h3 id="Closing-thoughts"><a href="#Closing-thoughts" class="headerlink" title="Closing thoughts"></a>Closing thoughts</h3><p>Though it seems to be limited compared to XHR, I think the current feature set is good enough for most of the cases. The simple api makes it beginner friendly and (future) native support means one less dependency to load.</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;Fetch is a much needed improvement over XHR, it simplifies making network requests by exposing an easy to use api and having promise supp
      
    
    </summary>
    
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
  </entry>
  
  <entry>
    <title>Using Optimizely with React</title>
    <link href="https://www.sheshbabu.com/posts/using-optimizely-with-react/"/>
    <id>https://www.sheshbabu.com/posts/using-optimizely-with-react/</id>
    <published>2016-07-31T08:34:00.000Z</published>
    <updated>2024-05-19T06:17:28.272Z</updated>
    
    <content type="html"><![CDATA[<p>Optimizely is an A/B testing tool used to test different variations of the same page/component to see which converts better.</p><p>Let’s say you have an ecommerce website with a product grid. The products in the grid currently contain only minimal information about it - name, picture, price and Buy button. Let’s say you have an hypothesis that adding ratings and other details such as weight/size etc would make more people click on the Buy button. But you’re concerned that adding too much information would clutter the UI and drive away customers. One way of solving this would be to do an A/B test between the existing UI (called as “Control” in A/B testing world) and the new UI with additional details (called “Variation”) and see which has more people clicking on the Buy button (“Conversion”). The traffic to the website is split into two - one group would always see the the Control and other would always see the Variation.</p><p>In Optimizely, the above is called an “Experiment” and both “Control” and “Variation” are the experiment’s “Variations”. Once you create an experiment and its variations, you’ll be shown an visual editor where you can customise the appearance of each variation by modifying/rearranging the elements in the page. The visual editor then translates those cutomisations into jQuery code called <code>Variation Code</code>. Depending on which variation an user is grouped into, the Optimizely library loads the appropriate <code>Variation Code</code> and thus displaying different UIs.</p><p>This workflow works well for static websites, but if the website is dynamic and uses React, then letting the <code>Variation Code</code> do arbitrary DOM manipulations doesn’t look like a good idea.</p><p>One solution is to create different components for each variation and using a container component to render the correct variation. But before that we need to know which variation the user belongs to and if the experiment is running (active) or not. Fortunately, Optimizely exposes <a href="http://developers.optimizely.com/javascript/reference/#the-data-object" target="_blank" rel="noopener">Data Object</a> which can be used to get the above data. We can use <code>window.optimizely.data.state.activeExperiments</code> to get the list of all running experiments and <code>window.optimizely.data.state.variationIdsMap[&lt;experimentId&gt;][0]</code> to get the variation the user belongs to.</p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// abtest-container.js</span><span class="token keyword">var</span> AbTestContainer <span class="token operator">=</span> React<span class="token punctuation">.</span><span class="token function">createClass</span><span class="token punctuation">(</span><span class="token punctuation">{</span>    propTypes<span class="token punctuation">:</span> <span class="token punctuation">{</span>        experimentId<span class="token punctuation">:</span> React<span class="token punctuation">.</span>PropTypes<span class="token punctuation">.</span>string<span class="token punctuation">.</span>isRequired<span class="token punctuation">,</span>        defaultVariationId<span class="token punctuation">:</span> React<span class="token punctuation">.</span>PropTypes<span class="token punctuation">.</span>string<span class="token punctuation">.</span>isRequired<span class="token punctuation">,</span>        variations<span class="token punctuation">:</span> React<span class="token punctuation">.</span>PropTypes<span class="token punctuation">.</span><span class="token function">arrayOf</span><span class="token punctuation">(</span>React<span class="token punctuation">.</span>PropTypes<span class="token punctuation">.</span><span class="token function">shape</span><span class="token punctuation">(</span><span class="token punctuation">{</span>            variationId<span class="token punctuation">:</span> React<span class="token punctuation">.</span>PropTypes<span class="token punctuation">.</span>string<span class="token punctuation">.</span>isRequired<span class="token punctuation">,</span>            component<span class="token punctuation">:</span> React<span class="token punctuation">.</span>PropTypes<span class="token punctuation">.</span>element<span class="token punctuation">.</span>isRequired        <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">)</span>    <span class="token punctuation">}</span><span class="token punctuation">,</span>    getVariation<span class="token punctuation">:</span> <span class="token keyword">function</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>        <span class="token comment" spellcheck="true">// Make sure you perform appropriate guard checks before using this in production!</span>        <span class="token keyword">var</span> activeExperiments <span class="token operator">=</span> window<span class="token punctuation">.</span>optimizely<span class="token punctuation">.</span>data<span class="token punctuation">.</span>state<span class="token punctuation">.</span>activeExperiments<span class="token punctuation">;</span>        <span class="token keyword">var</span> isExperimentActive <span class="token operator">=</span> _<span class="token punctuation">.</span><span class="token function">contains</span><span class="token punctuation">(</span>activeExperiments<span class="token punctuation">,</span> <span class="token keyword">this</span><span class="token punctuation">.</span>props<span class="token punctuation">.</span>experimentId<span class="token punctuation">)</span><span class="token punctuation">;</span>        <span class="token keyword">var</span> variation<span class="token punctuation">,</span> variationId<span class="token punctuation">;</span>        <span class="token keyword">if</span> <span class="token punctuation">(</span>isExperimentActive<span class="token punctuation">)</span> <span class="token punctuation">{</span>            variationId <span class="token operator">=</span> window<span class="token punctuation">.</span>optimizely<span class="token punctuation">.</span>data<span class="token punctuation">.</span>state<span class="token punctuation">.</span>variationIdsMap<span class="token punctuation">[</span>experimentId<span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">;</span>        <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>            variationId <span class="token operator">=</span> <span class="token keyword">this</span><span class="token punctuation">.</span>props<span class="token punctuation">.</span>defaultVariationId<span class="token punctuation">;</span>        <span class="token punctuation">}</span>        variation <span class="token operator">=</span> _<span class="token punctuation">.</span><span class="token function">findWhere</span><span class="token punctuation">(</span><span class="token keyword">this</span><span class="token punctuation">.</span>props<span class="token punctuation">.</span>variations<span class="token punctuation">,</span> <span class="token punctuation">{</span> variationId<span class="token punctuation">:</span> variationId <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>        <span class="token keyword">return</span> variation<span class="token punctuation">;</span>    <span class="token punctuation">}</span><span class="token punctuation">,</span>    render<span class="token punctuation">:</span> <span class="token keyword">function</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>        <span class="token keyword">return</span> <span class="token keyword">this</span><span class="token punctuation">.</span><span class="token function">getVariation</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>And this can be used as follows</p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// products-page.js</span><span class="token operator">...</span>render<span class="token punctuation">:</span> <span class="token keyword">function</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">var</span> variations <span class="token operator">=</span> <span class="token punctuation">[</span>        <span class="token punctuation">{</span> variationId<span class="token punctuation">:</span> <span class="token string">'111'</span><span class="token punctuation">,</span> component<span class="token punctuation">:</span> <span class="token operator">&lt;</span>ProductGrid<span class="token operator">/</span><span class="token operator">></span> <span class="token punctuation">}</span><span class="token punctuation">,</span>        <span class="token punctuation">{</span> variationId<span class="token punctuation">:</span> <span class="token string">'222'</span><span class="token punctuation">,</span> component<span class="token punctuation">:</span> <span class="token operator">&lt;</span>ProductGridWithAdditionalDetails<span class="token operator">/</span><span class="token operator">></span> <span class="token punctuation">}</span>    <span class="token punctuation">]</span><span class="token punctuation">;</span>    <span class="token keyword">return</span> <span class="token punctuation">(</span>        <span class="token operator">&lt;</span>AbTestContainer            experimentId<span class="token operator">=</span><span class="token string">'000'</span>            defaultVariationId<span class="token operator">=</span><span class="token string">'111'</span>            variations<span class="token operator">=</span><span class="token punctuation">{</span>variations<span class="token punctuation">}</span>        <span class="token operator">/</span><span class="token operator">></span>    <span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token operator">...</span></code></pre><p>The IDs for experiment and variations would be present in the visual editor page under “Options -&gt; Diagnostic Report”.</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;Optimizely is an A/B testing tool used to test different variations of the same page/component to see which converts better.&lt;/p&gt;
&lt;p&gt;Let’s
      
    
    </summary>
    
    
      <category term="React" scheme="https://www.sheshbabu.com/tags/React/"/>
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
      <category term="Optimizely" scheme="https://www.sheshbabu.com/tags/Optimizely/"/>
    
  </entry>
  
  <entry>
    <title>Guidelines to choose a JavaScript library</title>
    <link href="https://www.sheshbabu.com/posts/guidelines-to-choose-a-javascript-library/"/>
    <id>https://www.sheshbabu.com/posts/guidelines-to-choose-a-javascript-library/</id>
    <published>2016-06-12T05:38:45.000Z</published>
    <updated>2024-05-19T06:17:28.271Z</updated>
    
    <content type="html"><![CDATA[<h3 id="How-important-is-this"><a href="#How-important-is-this" class="headerlink" title="How important is this?"></a>How important is this?</h3><p>Picking the right JavaScript library is very important, especially during the beginning of a project. If we’re not careful with our decisions during the beginning, we will end up spending a lot of time later cleaning up the mess. The more tightly coupled the codebase and the dependency, the more careful we must be in selecting the right one. Even more so for frameworks - as our code practically lives inside them. Here are some of the things I look for in an external dependency:</p><h3 id="1-How-invested-are-the-core-contributors"><a href="#1-How-invested-are-the-core-contributors" class="headerlink" title="1) How invested are the core contributors?"></a>1) How invested are the core contributors?</h3><ul><li>Every opensource project has a couple of core contributors and sometimes a company behind it.</li><li>Finding out how they’re using the project would be a good indicator of their commitment. Are they using it in production and on revenue generating parts of the business? Example: Facebook is using React for Newsfeed, Instagram etc</li><li>This does not always apply as not all opensource projects have a commercial entity backing them.</li></ul><h3 id="2-How-widely-is-it-used"><a href="#2-How-widely-is-it-used" class="headerlink" title="2) How widely is it used?"></a>2) How widely is it used?</h3><ul><li>If it is widely used by others, then you would have access to a lot of tutorials, discussions about best practices, how-tos, StackOverflow answers etc.</li><li>Edge cases and bugs would have been detected early and bugfixes made.</li><li>Widely used libraries/frameworks would also help in hiring as there would be a good number of developers with that experience. Also, a good number of developers would be interested in joining your company to gain experience.</li><li>This can be found out by keeping your ear to the ground for what’s going on in the JavaScript community.</li></ul><h3 id="3-How-are-breaking-changes-introduced"><a href="#3-How-are-breaking-changes-introduced" class="headerlink" title="3) How are breaking changes introduced?"></a>3) How are breaking changes introduced?</h3><ul><li>The Web moves very fast and breaking changes are inevitable. Breaking changes might be done for deprecating bad ways of doing things, remedying poor architectural decisions made in the past, performance optimizations, availability of new browser features etc.</li><li>How they’re introduced makes a lot of difference - is this done gradually and incrementally? Is this communicated in advance by showing deprecation warnings?</li><li>Are any migration tools provided? Example: jQuery provides <a href="https://github.com/jquery/jquery-migrate" target="_blank" rel="noopener">jQuery migrate</a>, React provides <a href="https://github.com/reactjs/react-codemod" target="_blank" rel="noopener">React codemod</a> scripts for migrating to newer versions etc.</li><li>This ties into the “How invested are the core contributors?” question. If they’re using it for important projects then they would be very careful and systematic with breaking changes.</li></ul><h3 id="4-How-is-the-documentation"><a href="#4-How-is-the-documentation" class="headerlink" title="4) How is the documentation?"></a>4) How is the documentation?</h3><ul><li>A good documentation makes the library easy to use and helps avoid wasting time.</li><li>This depends on the project - libraries with simple and intuitive APIs can get away with minimal documentation whereas complicated libraries with hard to understand concepts need extensive documentation, examples, and tutorials.</li><li>Depending on how tightly coupled the library and the codebase is going to be, go through the library’s documentation to get a feel of it and try to build a quick proof-of-concept to see if all the current and future requirements could be easily implemented using it.</li></ul><h3 id="5-How-actively-is-it-being-developed"><a href="#5-How-actively-is-it-being-developed" class="headerlink" title="5) How actively is it being developed?"></a>5) How actively is it being developed?</h3><ul><li>Actively developed projects ensure that any new bugs are quickly fixed, new functionality added and PRs merged (all depending on priority and validity).</li><li>This also depends on the project as some projects are just “done” with nothing more to be added or fixed.</li></ul><h3 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h3><p>I understand that this is a very broad topic and there might be a lot more factors that need to be considered. I formed these guidelines based on my personal experience and learning from others.</p>]]></content>
    
    <summary type="html">
    
      With all the new libraries coming up each day in the JS ecosystem, I created some guidelines to help me choose a good library
    
    </summary>
    
    
      <category term="Development" scheme="https://www.sheshbabu.com/tags/Development/"/>
    
      <category term="JavaScript" scheme="https://www.sheshbabu.com/tags/JavaScript/"/>
    
  </entry>
  
  <entry>
    <title>Writing maintainable tests using the data driven approach</title>
    <link href="https://www.sheshbabu.com/posts/writing-maintainable-tests-using-the-data-driven-approach/"/>
    <id>https://www.sheshbabu.com/posts/writing-maintainable-tests-using-the-data-driven-approach/</id>
    <published>2016-06-06T22:49:08.000Z</published>
    <updated>2024-05-19T06:17:28.273Z</updated>
    
    <content type="html"><![CDATA[<p>One of the main considerations while writing automated tests is to make them more mainatainable. Tests shouldn’t get in the way of refactoring the source. This can be achieved by making the tests <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself" target="_blank" rel="noopener">DRY</a> and abstracting out repeating logic.</p><p>Certain logic like form validation, url parsing etc make tests unmaintainable by repeating the same tests again and again with different inputs and outputs.</p><p>Consider a simple example of validating username:</p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// validation.js</span><span class="token keyword">function</span> isValidUserName <span class="token punctuation">(</span>userName<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">if</span> <span class="token punctuation">(</span>userName<span class="token punctuation">.</span>isBlank <span class="token operator">||</span>        userName<span class="token punctuation">.</span>isLessThanTwoCharacters <span class="token operator">||</span>        userName<span class="token punctuation">.</span>hasSpecialCharacters <span class="token operator">||</span>        userName<span class="token punctuation">.</span>hasNumbers<span class="token punctuation">)</span> <span class="token punctuation">{</span>        <span class="token keyword">return</span> <span class="token boolean">false</span><span class="token punctuation">;</span>    <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>        <span class="token keyword">return</span> <span class="token boolean">true</span><span class="token punctuation">;</span>    <span class="token punctuation">}</span><span class="token punctuation">}</span></code></pre><p>The tests for the above function would be something like this:</p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// validation.test.js</span><span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'should reject blank input'</span><span class="token punctuation">,</span> <span class="token keyword">function</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">var</span> isValid <span class="token operator">=</span> <span class="token function">isValidUserName</span><span class="token punctuation">(</span><span class="token string">''</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token function">expect</span><span class="token punctuation">(</span>isValid<span class="token punctuation">)</span><span class="token punctuation">.</span>to<span class="token punctuation">.</span>be<span class="token punctuation">.</span><span class="token boolean">false</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'should reject input with less than two characters'</span><span class="token punctuation">,</span> <span class="token keyword">function</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">var</span> isValid <span class="token operator">=</span> <span class="token function">isValidUserName</span><span class="token punctuation">(</span><span class="token string">'a'</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token function">expect</span><span class="token punctuation">(</span>isValid<span class="token punctuation">)</span><span class="token punctuation">.</span>to<span class="token punctuation">.</span>be<span class="token punctuation">.</span><span class="token boolean">false</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'should reject input with special characters'</span><span class="token punctuation">,</span> <span class="token keyword">function</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">var</span> isValid <span class="token operator">=</span> <span class="token function">isValidUserName</span><span class="token punctuation">(</span><span class="token string">'abc#'</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token function">expect</span><span class="token punctuation">(</span>isValid<span class="token punctuation">)</span><span class="token punctuation">.</span>to<span class="token punctuation">.</span>be<span class="token punctuation">.</span><span class="token boolean">false</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'should reject input with numbers'</span><span class="token punctuation">,</span> <span class="token keyword">function</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token keyword">var</span> isValid <span class="token operator">=</span> <span class="token function">isValidUserName</span><span class="token punctuation">(</span><span class="token string">'ab3c'</span><span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token function">expect</span><span class="token punctuation">(</span>isValid<span class="token punctuation">)</span><span class="token punctuation">.</span>to<span class="token punctuation">.</span>be<span class="token punctuation">.</span><span class="token boolean">false</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>These kind of tests would be tedious to maintain for certain kind of logic.</p><p>A better way would be to use the <a href="https://en.wikipedia.org/wiki/Data-driven_testing" target="_blank" rel="noopener">data driven approach</a>. In this approach, we decouple the data and the actual testing code.</p><pre class=" language-javascript"><code class="language-javascript"> <span class="token comment" spellcheck="true">// validation.data.json</span> <span class="token punctuation">{</span>     <span class="token string">"username"</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>         <span class="token punctuation">{</span>             <span class="token string">"title"</span><span class="token punctuation">:</span> <span class="token string">"should reject blank input"</span><span class="token punctuation">,</span>             <span class="token string">"input"</span><span class="token punctuation">:</span> <span class="token string">""</span><span class="token punctuation">,</span>             <span class="token string">"isValid"</span><span class="token punctuation">:</span> <span class="token boolean">false</span>         <span class="token punctuation">}</span><span class="token punctuation">,</span>         <span class="token punctuation">{</span>             <span class="token string">"title"</span><span class="token punctuation">:</span> <span class="token string">"should reject input with less than two characters"</span><span class="token punctuation">,</span>             <span class="token string">"input"</span><span class="token punctuation">:</span> <span class="token string">"a"</span><span class="token punctuation">,</span>             <span class="token string">"isValid"</span><span class="token punctuation">:</span> <span class="token boolean">false</span>         <span class="token punctuation">}</span><span class="token punctuation">,</span>         <span class="token punctuation">{</span>             <span class="token string">"title"</span><span class="token punctuation">:</span> <span class="token string">"should reject input with special characters"</span><span class="token punctuation">,</span>             <span class="token string">"input"</span><span class="token punctuation">:</span> <span class="token string">"abc#"</span><span class="token punctuation">,</span>             <span class="token string">"isValid"</span><span class="token punctuation">:</span> <span class="token boolean">false</span>         <span class="token punctuation">}</span><span class="token punctuation">,</span>         <span class="token punctuation">{</span>             <span class="token string">"title"</span><span class="token punctuation">:</span> <span class="token string">"should reject input with numbers"</span><span class="token punctuation">,</span>             <span class="token string">"input"</span><span class="token punctuation">:</span> <span class="token string">"ab3c"</span><span class="token punctuation">,</span>             <span class="token string">"isValid"</span><span class="token punctuation">:</span> <span class="token boolean">false</span>         <span class="token punctuation">}</span>     <span class="token punctuation">]</span> <span class="token punctuation">}</span></code></pre><p>And modify our test file as</p><pre class=" language-javascript"><code class="language-javascript"><span class="token comment" spellcheck="true">// validation.test.js</span><span class="token keyword">var</span> data <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'validation.data'</span><span class="token punctuation">)</span><span class="token punctuation">;</span>data<span class="token punctuation">.</span>username<span class="token punctuation">.</span><span class="token function">forEach</span><span class="token punctuation">(</span><span class="token keyword">function</span> <span class="token punctuation">(</span>test<span class="token punctuation">)</span> <span class="token punctuation">{</span>    <span class="token function">it</span><span class="token punctuation">(</span>test<span class="token punctuation">.</span>title<span class="token punctuation">,</span> <span class="token keyword">function</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>        <span class="token keyword">var</span> isValid <span class="token operator">=</span> <span class="token function">isValidUserName</span><span class="token punctuation">(</span>test<span class="token punctuation">.</span>input<span class="token punctuation">)</span><span class="token punctuation">;</span>        <span class="token function">expect</span><span class="token punctuation">(</span>isValid<span class="token punctuation">)</span><span class="token punctuation">.</span>to<span class="token punctuation">.</span><span class="token function">equal</span><span class="token punctuation">(</span>test<span class="token punctuation">.</span>isValid<span class="token punctuation">)</span><span class="token punctuation">;</span>    <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre><p>This test can now be quickly refactored if needed. A nice bonus of using this approach is that by extracting the test data from actual test cases, the test scenarios can be understood in a glance.</p>]]></content>
    
    <summary type="html">
    
      Data Driven Testing helps making the tests more maintainable and readable
    
    </summary>
    
    
      <category term="Testing" scheme="https://www.sheshbabu.com/tags/Testing/"/>
    
      <category term="Maintainability" scheme="https://www.sheshbabu.com/tags/Maintainability/"/>
    
  </entry>
  
</feed>
