in

Coronavirus by Air: The spread of Covid-19 in the Middle East

An investigation by BBC News Arabic has found how one Iranian airline contributed to the spread of coronavirus around the Middle East.




in

Five-year-old caught driving parents' car in Utah

The boy said he was travelling to California to buy a Lamborghini.




in

Coronavirus: I got a life-changing opportunity in lockdown

Ana Carmona chronicled her month in quarantine with her family in NYC, including when she got some big news.




in

How the Covid-19 pandemic is threatening Africa’s wildlife

Park rangers in Africa say the closure of safari tourism is leading to an increase in poaching.




in

Coronavirus: Russian hospital staff 'working without masks'

As coronavirus spreads in the provinces, more and more health workers are getting sick - and dying.




in

ICYMI: Penguin chicks and new dining ideas

Some of the stories from around the world that you may have missed this week.




in

Ahmaud Arbery: Joggers out in solidarity with the killed 25-year-old

People have been dedicating their workouts to Ahmaud Arbery who was shot and killed while out jogging.






in

Design Kickoff Meetings

Posted here for posterity: Design kickoff meetings are like first dates that prepare you for an exciting relationship with a person who doesn’t exist.

The post Design Kickoff Meetings appeared first on Zeldman on Web & Interaction Design.




in

A panel on accessibility, design inclusion and ethics, hiring and retaining diverse talent, and landing a job in UX.

It’s one thing to seek diverse talent to add to your team, another to retain the people you’ve hired. Why do so many folks we bring in to add depth and breadth of experience to our design and business decision-making process end up leaving? Hear thoughtful, useful answers to this question and other mysteries of […]

The post A panel on accessibility, design inclusion and ethics, hiring and retaining diverse talent, and landing a job in UX. appeared first on Zeldman on Web & Interaction Design.




in

A kindness

Dino works six days a week as a porter in my apartment building, cleaning walls and floors, removing trash, distributing recyclables. He’s one of those essential workers who are suddenly on the front lines. We’ve always been friendly. I’ve been hibernating in my apartment for days, because it’s what we’re all supposed to do, and […]

The post A kindness appeared first on Zeldman on Web & Interaction Design.




in

Going viral

Finishing Week 4 with Coronavirus, heading into Week 5. I’m home—haven’t needed to go to the hospital, thank God—and my fever petered out last week. So all that’s left are cold and cough symptoms and a totally debilitating complete lack of energy. Oh, and lower back pain: a bad cough threw one side of my […]

The post Going viral appeared first on Zeldman on Web & Interaction Design.




in

Introducing the New React DevTools

We are excited to announce a new release of the React Developer Tools, available today in Chrome, Firefox, and (Chromium) Edge!

What’s changed?

A lot has changed in version 4! At a high level, this new version should offer significant performance gains and an improved navigation experience. It also offers full support for React Hooks, including inspecting nested objects.

Visit the interactive tutorial to try out the new version or see the changelog for demo videos and more details.

Which versions of React are supported?

react-dom

  • 0-14.x: Not supported
  • 15.x: Supported (except for the new component filters feature)
  • 16.x: Supported

react-native

  • 0-0.61: Not supported
  • 0.62: Will be supported (when 0.62 is released)

How do I get the new DevTools?

React DevTools is available as an extension for Chrome and Firefox. If you have already installed the extension, it should update automatically within the next couple of hours.

If you use the standalone shell (e.g. in React Native or Safari), you can install the new version from NPM:

npm install -g react-devtools@^4

Where did all of the DOM elements go?

The new DevTools provides a way to filter components from the tree to make it easier to navigate deeply nested hierarchies. Host nodes (e.g. HTML <div>, React Native <View>) are hidden by default, but this filter can be disabled:

How do I get the old version back?

If you are working with React Native version 60 (or older) you can install the previous release of DevTools from NPM:

npm install --dev react-devtools@^3

For older versions of React DOM (v0.14 or earlier) you will need to build the extension from source:

# Checkout the extension source
git clone https://github.com/facebook/react-devtools

cd react-devtools

# Checkout the previous release branch
git checkout v3

# Install dependencies and build the unpacked extension
yarn install
yarn build:extension

# Follow the on-screen instructions to complete installation

Thank you!

We’d like to thank everyone who tested the early release of DevTools version 4. Your feedback helped improve this initial release significantly.

We still have many exciting features planned and feedback is always welcome! Please feel free to open a GitHub issue or tag @reactjs on Twitter.




in

Preparing for the Future with React Prereleases

To share upcoming changes with our partners in the React ecosystem, we’re establishing official prerelease channels. We hope this process will help us make changes to React with confidence, and give developers the opportunity to try out experimental features.

This post will be most relevant to developers who work on frameworks, libraries, or developer tooling. Developers who use React primarily to build user-facing applications should not need to worry about our prerelease channels.

React relies on a thriving open source community to file bug reports, open pull requests, and submit RFCs. To encourage feedback, we sometimes share special builds of React that include unreleased features.

Because the source of truth for React is our public GitHub repository, it’s always been possible to build a copy of React that includes the latest changes. However it’s much easier for developers to install React from npm, so we occasionally publish prerelease builds to the npm registry. A recent example is the 16.7 alpha, which included an early version of the Hooks API.

We would like to make it even easier for developers to test prerelease builds of React, so we’re formalizing our process with three separate release channels.

Release Channels

The information in this post is also available on our Release Channels page. We will update that document whenever there are changes to our release process.

Each of React’s release channels is designed for a distinct use case:

  • Latest is for stable, semver React releases. It’s what you get when you install React from npm. This is the channel you’re already using today. Use this for all user-facing React applications.
  • Next tracks the master branch of the React source code repository. Think of these as release candidates for the next minor semver release. Use this for integration testing between React and third party projects.
  • Experimental includes experimental APIs and features that aren’t available in the stable releases. These also track the master branch, but with additional feature flags turned on. Use this to try out upcoming features before they are released.

All releases are published to npm, but only Latest uses semantic versioning. Prereleases (those in the Next and Experimental channels) have versions generated from a hash of their contents, e.g. 0.0.0-1022ee0ec for Next and 0.0.0-experimental-1022ee0ec for Experimental.

The only officially supported release channel for user-facing applications is Latest. Next and Experimental releases are provided for testing purposes only, and we provide no guarantees that behavior won’t change between releases. They do not follow the semver protocol that we use for releases from Latest.

By publishing prereleases to the same registry that we use for stable releases, we are able to take advantage of the many tools that support the npm workflow, like unpkg and CodeSandbox.

Latest Channel

Latest is the channel used for stable React releases. It corresponds to the latest tag on npm. It is the recommended channel for all React apps that are shipped to real users.

If you’re not sure which channel you should use, it’s Latest. If you’re a React developer, this is what you’re already using.

You can expect updates to Latest to be extremely stable. Versions follow the semantic versioning scheme. Learn more about our commitment to stability and incremental migration in our versioning policy.

Next Channel

The Next channel is a prerelease channel that tracks the master branch of the React repository. We use prereleases in the Next channel as release candidates for the Latest channel. You can think of Next as a superset of Latest that is updated more frequently.

The degree of change between the most recent Next release and the most recent Latest release is approximately the same as you would find between two minor semver releases. However, the Next channel does not conform to semantic versioning. You should expect occasional breaking changes between successive releases in the Next channel.

Do not use prereleases in user-facing applications.

Releases in Next are published with the next tag on npm. Versions are generated from a hash of the build’s contents, e.g. 0.0.0-1022ee0ec.

Using the Next Channel for Integration Testing

The Next channel is designed to support integration testing between React and other projects.

All changes to React go through extensive internal testing before they are released to the public. However, there are myriad environments and configurations used throughout the React ecosystem, and it’s not possible for us to test against every single one.

If you’re the author of a third party React framework, library, developer tool, or similar infrastructure-type project, you can help us keep React stable for your users and the entire React community by periodically running your test suite against the most recent changes. If you’re interested, follow these steps:

  • Set up a cron job using your preferred continuous integration platform. Cron jobs are supported by both CircleCI and Travis CI.
  • In the cron job, update your React packages to the most recent React release in the Next channel, using next tag on npm. Using the npm cli:

    npm update react@next react-dom@next

    Or yarn:

    yarn upgrade react@next react-dom@next
  • Run your test suite against the updated packages.
  • If everything passes, great! You can expect that your project will work with the next minor React release.
  • If something breaks unexpectedly, please let us know by filing an issue.

A project that uses this workflow is Next.js. (No pun intended! Seriously!) You can refer to their CircleCI configuration as an example.

Experimental Channel

Like Next, the Experimental channel is a prerelease channel that tracks the master branch of the React repository. Unlike Next, Experimental releases include additional features and APIs that are not ready for wider release.

Usually, an update to Next is accompanied by a corresponding update to Experimental. They are based on the same source revision, but are built using a different set of feature flags.

Experimental releases may be significantly different than releases to Next and Latest. Do not use Experimental releases in user-facing applications. You should expect frequent breaking changes between releases in the Experimental channel.

Releases in Experimental are published with the experimental tag on npm. Versions are generated from a hash of the build’s contents, e.g. 0.0.0-experimental-1022ee0ec.

What Goes Into an Experimental Release?

Experimental features are ones that are not ready to be released to the wider public, and may change drastically before they are finalized. Some experiments may never be finalized — the reason we have experiments is to test the viability of proposed changes.

For example, if the Experimental channel had existed when we announced Hooks, we would have released Hooks to the Experimental channel weeks before they were available in Latest.

You may find it valuable to run integration tests against Experimental. This is up to you. However, be advised that Experimental is even less stable than Next. We do not guarantee any stability between Experimental releases.

How Can I Learn More About Experimental Features?

Experimental features may or may not be documented. Usually, experiments aren’t documented until they are close to shipping in Next or Stable.

If a feature is not documented, they may be accompanied by an RFC.

We will post to the React blog when we’re ready to announce new experiments, but that doesn’t mean we will publicize every experiment.

You can always refer to our public GitHub repository’s history for a comprehensive list of changes.




in

Building Great User Experiences with Concurrent Mode and Suspense

At React Conf 2019 we announced an experimental release of React that supports Concurrent Mode and Suspense. In this post we’ll introduce best practices for using them that we’ve identified through the process of building the new facebook.com.

This post will be most relevant to people working on data fetching libraries for React.

It shows how to best integrate them with Concurrent Mode and Suspense. The patterns introduced here are based on Relay — our library for building data-driven UIs with GraphQL. However, the ideas in this post apply to other GraphQL clients as well as libraries using REST or other approaches.

This post is aimed at library authors. If you’re primarily an application developer, you might still find some interesting ideas here, but don’t feel like you have to read it in its entirety.

Talk Videos

If you prefer to watch videos, some of the ideas from this blog post have been referenced in several React Conf 2019 presentations:

This post presents a deeper dive on implementing a data fetching library with Suspense.

Putting User Experience First

The React team and community has long placed a deserved emphasis on developer experience: ensuring that React has good error messages, focusing on components as a way to reason locally about app behavior, crafting APIs that are predictable and encourage correct usage by design, etc. But we haven’t provided enough guidance on the best ways to achieve a great user experience in large apps.

For example, the React team has focused on framework performance and providing tools for developers to debug and tune application performance (e.g. React.memo). But we haven’t been as opinionated about the high-level patterns that make the difference between fast, fluid apps and slow, janky ones. We always want to ensure that React remains approachable to new users and supports a variety of use-cases — not every app has to be “blazing” fast. But as a community we can and should aim high. We should make it as easy as possible to build apps that start fast and stay fast, even as they grow in complexity, for users on varying devices and networks around the world.

Concurrent Mode and Suspense are experimental features that can help developers achieve this goal. We first introduced them at JSConf Iceland in 2018, intentionally sharing details very early to give the community time to digest the new concepts and to set the stage for subsequent changes. Since then we’ve completed related work, such as the new Context API and the introduction of Hooks, which are designed in part to help developers naturally write code that is more compatible with Concurrent Mode. But we didn’t want to implement these features and release them without validating that they work. So over the past year, the React, Relay, web infrastructure, and product teams at Facebook have all collaborated closely to build a new version of facebook.com that deeply integrates Concurrent Mode and Suspense to create an experience with a more fluid and app-like feel.

Thanks to this project, we’re more confident than ever that Concurrent Mode and Suspense can make it easier to deliver great, fast user experiences. But doing so requires rethinking how we approach loading code and data for our apps. Effectively all of the data-fetching on the new facebook.com is powered by Relay Hooks — new Hooks-based Relay APIs that integrate with Concurrent Mode and Suspense out of the box.

Relay Hooks — and GraphQL — won’t be for everyone, and that’s ok! Through our work on these APIs we’ve identified a set of more general patterns for using Suspense. Even if Relay isn’t the right fit for you, we think the key patterns we’ve introduced with Relay Hooks can be adapted to other frameworks.

Best Practices for Suspense

It’s tempting to focus only on the total startup time for an app — but it turns out that users’ perception of performance is determined by more than the absolute loading time. For example, when comparing two apps with the same absolute startup time, our research shows that users will generally perceive the one with fewer intermediate loading states and fewer layout changes as having loaded faster. Suspense is a powerful tool for carefully orchestrating an elegant loading sequence with a few, well-defined states that progressively reveal content. But improving perceived performance only goes so far — our apps still shouldn’t take forever to fetch all of their code, data, images, and other assets.

The traditional approach to loading data in React apps involves what we refer to as “fetch-on-render”. First we render a component with a spinner, then fetch data on mount (componentDidMount or useEffect), and finally update to render the resulting data. It’s certainly possible to use this pattern with Suspense: instead of initially rendering a placeholder itself, a component can “suspend” — indicate to React that it isn’t ready yet. This will tell React to find the nearest ancestor <Suspense fallback={<Placeholder/>}>, and render its fallback instead. If you watched earlier Suspense demos this example may feel familiar — it’s how we originally imagined using Suspense for data-fetching.

It turns out that this approach has some limitations. Consider a page that shows a social media post by a user, along with comments on that post. That might be structured as a <Post> component that renders both the post body and a <CommentList> to show the comments. Using the fetch-on-render approach described above to implement this could cause sequential round trips (sometimes referred to as a “waterfall”). First the data for the <Post> component would be fetched and then the data for <CommentList> would be fetched, increasing the time it takes to show the full page.

There’s also another often-overlooked downside to this approach. If <Post> eagerly requires (or imports) the <CommentList> component, our app will have to wait to show the post body while the code for the comments is downloading. We could lazily load <CommentList>, but then that would delay fetching comments data and increase the time to show the full page. How do we resolve this problem without compromising on the user experience?

Render As You Fetch

The fetch-on-render approach is widely used by React apps today and can certainly be used to create great apps. But can we do even better? Let’s step back and consider our goal.

In the above <Post> example, we’d ideally show the more important content — the post body — as early as possible, without negatively impacting the time to show the full page (including comments). Let’s consider the key constraints on any solution and look at how we can achieve them:

  • Showing the more important content (the post body) as early as possible means that we need to load the code and data for the view incrementally. We don’t want to block showing the post body on the code for <CommentList> being downloaded, for example.
  • At the same time we don’t want to increase the time to show the full page including comments. So we need to start loading the code and data for the comments as soon as possible, ideally in parallel with loading the post body.

This might sound difficult to achieve — but these constraints are actually incredibly helpful. They rule out a large number of approaches and spell out a solution for us. This brings us to the key patterns we’ve implemented in Relay Hooks, and that can be adapted to other data-fetching libraries. We’ll look at each one in turn and then see how they add up to achieve our goal of fast, delightful loading experiences:

  1. Parallel data and view trees
  2. Fetch in event handlers
  3. Load data incrementally
  4. Treat code like data

Parallel Data and View Trees

One of the most appealing things about the fetch-on-render pattern is that it colocates what data a component needs with how to render that data. This colocation is great — an example of how it makes sense to group code by concerns and not by technologies. All the issues we saw above were due to when we fetch data in this approach: upon rendering. We need to be able to fetch data before we’ve rendered the component. The only way to achieve that is by extracting the data dependencies into parallel data and view trees.

Here’s how that works in Relay Hooks. Continuing our example of a social media post with body and comments, here’s how we might define it with Relay Hooks:

// Post.js
function Post(props) {
  // Given a reference to some post - `props.post` - *what* data
  // do we need about that post?
  const postData = useFragment(graphql`
    fragment PostData on Post @refetchable(queryName: "PostQuery") {
      author
      title
      # ...  more fields ...
    }
  `, props.post);

  // Now that we have the data, how do we render it?
  return (
    <div>
      <h1>{postData.title}</h1>
      <h2>by {postData.author}</h2>
      {/* more fields  */}
    </div>
  );
}

Although the GraphQL is written within the component, Relay has a build step (Relay Compiler) that extracts these data-dependencies into separate files and aggregates the GraphQL for each view into a single query. So we get the benefit of colocating concerns, while at runtime having parallel data and view trees. Other frameworks could achieve a similar effect by allowing developers to define data-fetching logic in a sibling file (maybe Post.data.js), or perhaps integrate with a bundler to allow defining data dependencies with UI code and automatically extracting it, similar to Relay Compiler.

The key is that regardless of the technology we’re using to load our data — GraphQL, REST, etc — we can separate what data to load from how and when to actually load it. But once we do that, how and when do we fetch our data?

Fetch in Event Handlers

Imagine that we’re about to navigate from a list of a user’s posts to the page for a specific post. We’ll need to download the code for that page — Post.js — and also fetch its data.

Waiting until we render the component has problems as we saw above. The key is to start fetching code and data for a new view in the same event handler that triggers showing that view. We can either fetch the data within our router — if our router supports preloading data for routes — or in the click event on the link that triggered the navigation. It turns out that the React Router folks are already hard at work on building APIs to support preloading data for routes. But other routing frameworks can implement this idea too.

Conceptually, we want every route definition to include two things: what component to render and what data to preload, as a function of the route/url params. Here’s what such a route definition might look like. This example is loosely inspired by React Router’s route definitions and is primarily intended to demonstrate the concept, not a specific API:

// PostRoute.js (GraphQL version)

// Relay generated query for loading Post data
import PostQuery from './__generated__/PostQuery.graphql';

const PostRoute = {
  // a matching expression for which paths to handle
  path: '/post/:id',

  // what component to render for this route
  component: React.lazy(() => import('./Post')),

  // data to load for this route, as function of the route
  // parameters
  prepare: routeParams => {
    // Relay extracts queries from components, allowing us to reference
    // the data dependencies -- data tree -- from outside.
    const postData = preloadQuery(PostQuery, {
      postId: routeParams.id,
    });

    return { postData };
  },
};

export default PostRoute;

Given such a definition, a router can:

  • Match a URL to a route definition.
  • Call the prepare() function to start loading that route’s data. Note that prepare() is synchronous — we don’t wait for the data to be ready, since we want to start rendering more important parts of the view (like the post body) as quickly as possible.
  • Pass the preloaded data to the component. If the component is ready — the React.lazy dynamic import has completed — the component will render and try to access its data. If not, React.lazy will suspend until the code is ready.

This approach can be generalized to other data-fetching solutions. An app that uses REST might define a route like this:

// PostRoute.js (REST version)

// Manually written logic for loading the data for the component
import PostData from './Post.data';

const PostRoute = {
  // a matching expression for which paths to handle
  path: '/post/:id',

  // what component to render for this route
  component: React.lazy(() => import('./Post')),

  // data to load for this route, as function of the route
  // parameters
  prepare: routeParams => {
    const postData = preloadRestEndpoint(
      PostData.endpointUrl, 
      {
        postId: routeParams.id,
      },
    );
    return { postData };
  },
};

export default PostRoute;

This same approach can be employed not just for routing, but in other places where we show content lazily or based on user interaction. For example, a tab component could eagerly load the first tab’s code and data, and then use the same pattern as above to load the code and data for other tabs in the tab-change event handler. A component that displays a modal could preload the code and data for the modal in the click handler that triggers opening the modal, and so on.

Once we’ve implemented the ability to start loading code and data for a view independently, we have the option to go one step further. Consider a <Link to={path} /> component that links to a route. If the user hovers over that link, there’s a reasonable chance they’ll click it. And if they press the mouse down, there’s an even better chance that they’ll complete the click. If we can load code and data for a view after the user clicks, we can also start that work before they click, getting a head start on preparing the view.

Best of all, we can centralize that logic in a few key places — a router or core UI components — and get any performance benefits automatically throughout our app. Of course preloading isn’t always beneficial. It’s something an application would tune based on the user’s device or network speed to avoid eating up user’s data plans. But the pattern here makes it easier to centralize the implementation of preloading and the decision of whether to enable it or not.

Load Data Incrementally

The above patterns — parallel data/view trees and fetching in event handlers — let us start loading all the data for a view earlier. But we still want to be able to show more important parts of the view without waiting for all of our data. At Facebook we’ve implemented support for this in GraphQL and Relay in the form of some new GraphQL directives (annotations that affect how/when data is delivered, but not what data). These new directives, called @defer and @stream, allow us to retrieve data incrementally. For example, consider our <Post> component from above. We want to show the body without waiting for the comments to be ready. We can achieve this with @defer and <Suspense>:

// Post.js
function Post(props) {
  const postData = useFragment(graphql`
    fragment PostData on Post {
      author
      title

      # fetch data for the comments, but don't block on it being ready
      ...CommentList @defer
    }
  `, props.post);

  return (
    <div>
      <h1>{postData.title}</h1>
      <h2>by {postData.author}</h2>
      {/* @defer pairs naturally with <Suspense> to make the UI non-blocking too */}
      <Suspense fallback={<Spinner/>}>
        <CommentList post={postData} />
      </Suspense>
    </div>
  );
}

Here, our GraphQL server will stream back the results, first returning the author and title fields and then returning the comment data when it’s ready. We wrap <CommentList> in a <Suspense> boundary so that we can render the post body before <CommentList> and its data are ready. This same pattern can be applied to other frameworks as well. For example, apps that call a REST API might make parallel requests to fetch the body and comments data for a post to avoid blocking on all the data being ready.

Treat Code Like Data

But there’s one thing that’s still missing. We’ve shown how to preload data for a route — but what about code? The example above cheated a bit and used React.lazy. However, React.lazy is, as the name implies, lazy. It won’t start downloading code until the lazy component is actually rendered — it’s “fetch-on-render” for code!

To solve this, the React team is considering APIs that would allow bundle splitting and eager preloading for code as well. That would allow a user to pass some form of lazy component to a router, and for the router to trigger loading the code alongside its data as early as possible.

Putting It All Together

To recap, achieving a great loading experience means that we need to start loading code and data as early as possible, but without waiting for all of it to be ready. Parallel data and view trees allow us to load the data for a view in parallel with loading the view (code) itself. Fetching in an event handler means we can start loading data as early as possible, and even optimistically preload a view when we have enough confidence that a user will navigate to it. Loading data incrementally allows us to load important data earlier without delaying the fetching of less important data. And treating code as data — and preloading it with similar APIs — allows us to load it earlier too.

Using These Patterns

These patterns aren’t just ideas — we’ve implemented them in Relay Hooks and are using them in production throughout the new facebook.com (which is currently in beta testing). If you’re interested in using or learning more about these patterns, here are some resources:

  • The React Concurrent docs explore how to use Concurrent Mode and Suspense and go into more detail about many of these patterns. It’s a great resource to learn more about the APIs and use-cases they support.
  • The experimental release of Relay Hooks implements the patterns described here.
  • We’ve implemented two similar example apps that demonstrate these concepts:

    • The Relay Hooks example app uses GitHub’s public GraphQL API to implement a simple issue tracker app. It includes nested route support with code and data preloading. The code is fully commented — we encourage cloning the repo, running the app locally, and exploring how it works.
    • We also have a non-GraphQL version of the app that demonstrates how these concepts can be applied to other data-fetching libraries.

While the APIs around Concurrent Mode and Suspense are still experimental, we’re confident that the ideas in this post are proven by practice. However, we understand that Relay and GraphQL aren’t the right fit for everyone. That’s ok! We’re actively exploring how to generalize these patterns to approaches such as REST, and are exploring ideas for a more generic (ie non-GraphQL) API for composing a tree of data dependencies. In the meantime, we’re excited to see what new libraries will emerge that implement the patterns described in this post to make it easier to build great, fast user experiences.




in

An intro to making Postgres high availability on Kubernetes

#351 — April 15, 2020

Read on the Web

Postgres Weekly

A Detailed Look at pg_show_plans — A few issues ago we linked to a basic introduction to pg_show_plans – this goes a little further. pg_show_plans lets you look at the execution plans of slow queries in real time as they’re being executed which can help you when troubleshooting.

Kaarel Moppel

Intersecting GPS Tracks to Identify Infected Individuals — I’m not a huge fan of COVID-19 related content, but this is a pretty interesting technique with numerous use cases. Essentially it uses PostGIS to identify overlapping paths.

Florian Nadler

Online Training: Learn PostgreSQL from Home — The remote PostgreSQL Database Administration training course is available at a discounted rate & will be conducted in two different timezones. The course covers day-to-day DBA operations, monitoring, server configurations, and more.

2ndQuadrant PostgreSQL Training sponsor

PostgreSQL's 'Related Projects' — Thanks to Andreas Scherbaum for pointing out a new page on the Postgres site dedicated to projects related to Postgres like the code that runs the Postgres web site, mailing list, build farm, package management system, etc.

PostgreSQL Global Development Group

Authentication Configuration in Postgres (and CockroachDB) — In Postgres, client authentication can be controlled via a ‘HBA’ (host-based authentication) file. It’s not something we see covered very often, so you might find this interesting, particularly as it compares things against CockroachDB.

Raphael ‘kena’ Poss

▶  Easy And Correct High Availability Postgres with Kubernetes — A 50 minute talk from PostgresOpen 2019 that goes all the way ‘from containers up’ until actually doing stuff with Postgres.

Steven Pousty

How To Set Up an Express API Backend Project With Postgres — A pretty extensive walkthrough of creating an HTTP API using Express with Node.js and Postgres on the backend, then deploying it all on Heroku.

Chidi Orji

A Beginners Guide to Basic Indexing in Postgres

James Bannister

eBook: The Most Important Events to Monitor in Your Postgres Logs — In this eBook, we are looking at the Top 6 Postgres log events for monitoring query performance and preventing downtime.

pganalyze sponsor

Documenting the Citus Extension to Postgres: An Interview with Joe Nelson — Joe, a.k.a. begriffs, talks about why he works on documentation, why the multi-tenant and real-time analytics tutorials matter, the INSERT..SELECT with repartitioning feature, and what development platform Citus uses for docs.

Citus Data (Microsoft)

Procedural vs Query Approaches for Finding Packages — Explorations of a query that can be used to display which packages are available for a given FreeBSD port. Get your head around the data model and the ideas here apply to all sorts of situations.

Dan Langille

???? Upcoming Events

All in-person events we had listed are cancelled or postponed due to the COVID outbreak, so we're now linking to webinars, livestreams, and similar online events.

If you have any, just hit reply and if it's Postgres related (and either free or not too expensive) we'll include it in a future issue. Just one this week:

???? – requires e-mail address or registration
???? – costs money to participate

???? Seen on Twitter

Saw this tweet and thought it was a pretty neat reminder of the sorts of things we can do with Postgres. Justin kindly let us include it:

Click through to the original tweet if you want to see the code better. Neat use for a generated column!




in

The one where we interviewed Luca Ferrari

#352 — April 22, 2020

Read on the Web

???? We've got a neat bonus for you this week at the bottom of the issue ????

Postgres Weekly

Postgres Explain Visualizer 2: A Vue.js Component to Show Execution Plans — Less a standalone tool and something you’d use when building your own Postgres tooling. There is, however, a demo here. The output is really nifty.

Dalibo

Insert-Only Tables To Be Autovacuumed in Postgres 13 (But Why?) — Autovacuuming clears up dead tuples that are often left when updating or deleting data from tables, so why is autovacuuming for append-only tables a big deal in Postgres 13? Laurenz explains.

Laurenz Albe

Faster CI/CD for All Your Software Projects Using Buildkite — See how Shopify scaled from 300 to 1800 engineers while keeping their build times under 5 minutes.

Buildkite sponsor

Full Text Search in Milliseconds with Rails and Postgres — If you’ve never played with full text search with Postgres and Rails, this is a fine place to start. It covers LIKE/ILIKE, trigrams, and ‘proper’ full text searching. We also get to see how Leigh took a query from taking 130ms down to 7ms.

Leigh Halliday

An Easy Postgres 12 and pgAdmin 4 Setup with Docker — Docker provides an easy and loosely coupled way to get things set up in a development environment.

Jonathan S. Katz

Is There a Limit on Number of Partitions Handled by Postgres? — Sort of, but you’d really have to be going at it to stretch Postgres 12’s capabilities in this area.

Denish Patel

Where Do My Postgres Settings Come From? — A nice visual look at how parameters and settings cascade or override each other.

My DBA Notebook

Identify Slow-Running PostgreSQL Queries Quickly in Datadog — Improve PostgreSQL performance by visualizing and identifying errors fast using granular, out-of-the-box dashboards in Datadog.

Datadog sponsor

Replicate Multiple Postgres Servers to a Single MongoDB Server using Logical Decoding Output Plugin

David Zhang

An Overview of the JOIN Methods in Postgres

Kumar Rajeev Rastogi

???? A Q&A with…
Luca Ferrari
Postgres community organizer, author, adjunct professor, and open source advocate.

Luca Ferrari has had a huge impact on the Postgres community in Italy, having been president of the Italian PostgreSQL Users Group in the past and having helped to organize the popular PGDay.it events. He also blogs frequently about Postgres and wrote PostgreSQL 11 Server Side Programming Quick Start Guide for Packt.

Note: A more complete version of this interview is on the Web.

We caught up with him to ask about server side Postgres use cases in particular:

For those who use Postgres as a simple database and haven't touched the deeper elements, where do you think they should start?

There's no single answer to this question, since Postgres is such a huge project with so many features and a rich community. I never found a project where it cannot fit in. Postgres is somehow like Unix: you cannot touch it as "just a database", you need to commit to its culture to benefit the most out of it.

In my classes, I can see that people usually get fascinated by the capabilities of doing server side programming, and that is why I decided to write my book about this topic. Often, people do not expect to be able to embed their Perl, Java, or Python libraries directly into PostgreSQL without having to rewrite their business logic in an SQL-like language.

Another great feature nowdays is the support to JSON within the database, thanks to which PostgreSQL can be used as both a relational database and a 'NoSQL' storage engine, providing a lot of flexibility in your infrastructure.

One suggestion I always gave is to join the mailing lists: there are several that differ by topic and amount of traffic. Most are very active and have high quality contributors that take care in providing accuate replies to users' questions, that spend time in reproducing errors and edge-cases, and who will help you. That's a mandatory place where you have to start, in my opinion, to better learn about the project, its features and its culture.

Where should the line be drawn between doing things in an external programming language versus within Postgres?

Often the right choice is to place business logic near the data it refers to, that is within the database itself. However, there are several things to take into account including the developer's experience and the expressiveness of SQL-derived languages like pl/PgSQL.

There's a habit of letting ORMs (Object Relational Mappers) do most of the database interaction nowadays, reducing the database to a "simple storage". Of course, databases can do a lot more, and PostgreSQL in particular can help you migrate and embed your own business logic into the database itself.

I have helped a few companies embed their own Java libraries into Postgres resulting in a more robust and coherent way to access the data (the real value) without any regard of the application they were using. Because once you start having data, you will soon find that such data is required by multiple applications in different technologies and on different platforms, so that implementing the same business logic rules over and over becomes a huge effort; on the other hand, moving such logic within the database simplifies and keep uniform the way your data is manipulated.

What one thing do you think people should learn?

Stored procedures. They serve as a common base for triggers and are very similar to routines, therefore allowing you to build more complex pieces into your own cluster. Once you have learnt the common way of defining functions, you can go deeper and write your own native functions using other languages (e.g. C). This is more complex, but thanks to the extensibility of Postgres is not an impossible task and can help you migrating more and more code into the database. Once you have created a new feature, please do contribute it back so that other people can use it!

...

Last, allow me to announce that I'm working on another book right now: me and a friend of mine are writing a more general book on Postgres that will try to answer your question by leading the reader through the main features that make Postgres unique and great.

Be sure to check out Luca's book PostgreSQL 11 Server Side Programming Quick Start Guide to learn more about the topics covered in this interview. You can also find the code from the book in this GitHub repo.

If you enjoyed this interview, Luca actually gave some more detailed answers in the full interview which you can read here.




in

Workloads, acceleration, and making Postgres better

#353 — April 29, 2020

Read on the Web

Postgres Weekly

7 Things That Could Be Improved in Postgres — As 1990s dance pop group D:Ream sang in 1994, Things Can Only Get Better.. including Postgres ???? Luckily these are all ‘nice to have’s but I dare say we’ll see some of them (such as automatic tuning and auto-vacuuming improvements) appear over time.

Kaarel Moppel

How The Citus Distributed Query Executor Adapts to a Postgres Workload — Citus is the popular extension for horizontally scaling Postgres and its query executor has seen some huge updates lately.

Citus Data (Microsoft)

eBook: The Most Important Events to Monitor in Your Postgres Logs — In this pganalyze eBook, we are looking at the Top 6 Postgres log events for monitoring query performance and preventing downtime.

pganalyze sponsor

Swarm64 DA 4.0: A Database Acceleration Extension for Postgres — Swarm64 started life as a FPGA-driven way to accelerate Postgres performance, but can now work without FPGAs too. This is not a free product but if you want to give it a run, there’s a trial or it can be spun up from the AWS Marketplace.

Yana Krasteva

Postgres Performance Goalposts — An interesting heuristic from Bruce here on what to do if you expect your connections, queries, or write queries to be above/below certain levels.

Bruce Momjian

A Tale of Password Authentication Methods in Postgres“Let’s say you want to implement a password authentication method in a client/server protocol..” Here’s the story of how Postgres came up with its approaches.

Peter Eisentraut

How to Set application_name When Using psql — As Craig says: “Setting your application name in Postgres is SO USEFUL. It will help a lot for debugging when you’ve got multiple different apps/services connecting to the same database.”

Denish Patel

How to Upgrade Postgres from v11 to v12 on Ubuntu 20.04 — Now that Ubuntu 20.04 is out, this might be on your mind!

Paolo Melchiorre

Working with Amazon Aurora PostgreSQL: What Happened to the Stats? — Apparently there’s a bug with numerous versions of Aurora PostgreSQL that causes certain stats to be lost on restart.

Michael Vitale

Postgres Vision 2020 - Free Online Conference (June 23-24) — Learn how today’s IT leaders are using Postgres. Join from anywhere in the world and listen from 30+ Postgres experts.

EnterpriseDB sponsor

A Deep Dive into PostGIS Nearest Neighbor Search — Take a deep dive into the Postgres and PostGIS internals to find out how K-nearest neighbor accelerates local search.

Martin Davis

My Favorite Postgres Extensions: Part One — A basic high level look at pg_partman and postgres_fdw.

Nawaz Ahmed

Kanel: Generate TypeScript Types from Postgres

Kristian Dupont

Postgres.app: The Easiest Way to Get Started with Postgres on the Mac — I’ve used this for years, it’s super popular, but if there’s just a handful of developers out there who’d benefit from it and don’t know about it, this reminder will be worth it :-) It continues to get very frequent updates.

Jakob Egger, Chris Pastl, and Mattt Thompson

???? Upcoming Events

All in-person events we had listed are cancelled or postponed due to the COVID outbreak, so we're now linking to webinars, livestreams, and similar online events.

If you have any, just hit reply and if it's Postgres related (and either free or not too expensive) we'll include it in a future issue. Just one this week:

  • ???? Postgres Vision 2020 on June 23-24. A full attempt at an online Postgres conference across multiple days with multiple tracks.

???? – requires e-mail address or registration
???? – costs money to participate




in

Medium-hard SQL questions to think about

#354 — May 6, 2020

Read on the Web

Postgres Weekly

pgModeler: A Postgres Database Modeler — An easy way to create and edit database models in a visual way. It’s packaged up as a paid product but is also open source so you can build your own.

Raphael Araújo e Silva

The Best Medium-Hard Data Analyst SQL Interview Questions — This article begins with a quote: “The first 70% of SQL is pretty straightforward but the remaining 30% can be pretty tricky.” True! This article focuses on the tricky ‘medium-hard’ area that few tutorials venture into.

Zachary Thomas

Monitor Custom Postgres Metrics in Real-Time with Datadog — Monitor and visualize Postgres performance in context end-to-end alongside the rest of your stack. Create custom, drag-and-drop dashboards to quickly view analytics on any Postgres metric. Try Datadog free.

Datadog sponsor

My Favorite PostgreSQL Extensions: Part Two — The second part of a series we linked to last week. This time, Nawaz takes a look at pgAudit, pg_repack, and HypoPG.

Nawaz Ahmed

Backup Manifests and pg_verifybackup in Postgres 13 — Postgres 13 will introduce two features to enhance the automated validation of physical backups: backup manifests and a new tool called pg_verifybackup.

Gabriele Bartolini

arm64 Packages Now on apt.postgresql.org — If you’re running ARM64 hardware and Debian or Ubuntu, you can now install Postgres via apt.

Christoph Berg

Speeding Up count(*): Why Not Use max(id) - min(id)? — A warning tale in case you decide to take this shortcut. While you might be able to estimate or fudge a number that’s close, you can’t guarantee sequences will give you an exact, correct answer here.

Hans-Jürgen Schönig

Using Postgres for JSON Storage — With JSON and JSONB types and associated advanced ways to query such columns, using Postgres as a store for JSON data is pretty simple. This is the briefest of overviews but leads into an interactive online tutorial.

Steve Pousty

How to Migrate From Inheritance-Based Partitioning to Declarative Partitioning — Partitioning was introduced in Postgres 10 and Postgres 11 improved the declarative partitioning support. This article demonstrates a move from inheritance based partitioning to declarative partitioning using the native features found in Postgres 11+.

Caterina Magini

Free eBook: How to Get a 3x Performance Improvement on Your Postgres Database — Learn our best practices for optimizing Postgres query performance for customers like Atlassian and how to reduce data loaded from disk by 500x.

pganalyze sponsor

How to Backup Multiple Tablespaces with pg_basebackup

Ahsan Hadi

▶  Security and Compliance with Postgres — A recorded webinar that 2ndQuadrant ran recently.

Boriss Mejías

Oracle to PostgreSQL: ANSI Outer Join Syntax in Postgres — The latest in a series of blog posts about migrating to Postgres from Oracle which looks at what Postgres offers in place of Oracle’s join operators.

Kirk Roybal

An Interview with 2ndQuadrant's Jimmy Angelakos — The latest ‘PostgreSQL Person of the Week’ to face questions about his experiences with Postgres.

Andreas Scherbaum

dadbod.vim: A Modern Database Interface for Vim — A Vim plugin for interacting with numerous databases, including Postgres.

Tim Pope

???? Upcoming Online Events

  • Postgres Pulse - weekly at 11am ET each Monday. Weekly Zoom-based sessions with folks like Bruce Momjian, Vibhor Kumar, and other people at EnterpriseDB.
  • ???? Postgres Vision 2020 on June 23-24. A full attempt at an online Postgres conference across multiple days with multiple tracks.

???? – requires e-mail address or registration
???? – costs money to participate




in

Stimulus Reflex, and sending thanks to Matz

#498 — April 23, 2020

Read on the Web

Ruby Weekly

Credit: Divina Epiphania / Shutterstock.com

Mining for Malicious Ruby Gems: 700+ Gems Affected — Breathe easy as this was all resolved a month ago (and was too obscure to pay off for the hackers anyway) but a security research team recently found over 700 malicious Ruby gems that were subtle typos/adjustments of more popular gems (e.g. atlas-client vs atlas_client – could you tell which one is real?)

Tomislav Maljic

You Can Now Sponsor Matz on GitHub — I appreciate these are challenging times, but if you’ve ever wanted to give a big thank you to Matz, the creator of Ruby, here’s one way to do it. We’re sponsoring Matz now as without him, this newsletter wouldn’t exist! ???? Alternatively, if you have little to spare, maybe send him a thanks on Twitter?

GitHub Sponsors

Ruby Performance Tips — Here’s a collection of practical tips for improving Ruby performance for better user experiences, brought to you by Raygun. Read the tips here.

Raygun sponsor

Full Text Search in Milliseconds with Rails and Postgres — If you’ve never played with full text search with Postgres and Rails, this is a fine place to start. It covers LIKE/ILIKE, trigrams, and ‘proper’ full text searching. We also get to see how Leigh took a query from taking 130ms down to 7ms.

Leigh Halliday

▶  Introduction to Stimulus Reflex — Stimiulus Reflex makes SPA-type interactions very simple by using ActionCable to render pages and then diffing them on the client.

GoRails

Rails Performance: When is Caching the Right Choice? — Before you say “always”, understand that caching is not free and, if done incorrectly, can even make things worse.

Jonathan Miles

???? Jobs

Find a Job Through Vettery — Vettery specializes in tech roles and is completely free for job seekers. Create a profile to get started.

Vettery

Ruby Backend Developer (Austria) — We’re seeking mid-level and senior devs to join us and build top-class backend infrastructure for our adidas apps, used by millions. Our stack includes: jRuby, Sinatra, Sidekiq, MySQL, & MongoDB.

Runtastic

ℹ️ Interested in running a job listing in Ruby Weekly? There's more info here.

???? Articles & Tutorials

How to Customize Webpack in Rails Apps — How to go about configuring webpack when tweaking webpacker.yml just isn’t enough.

Ross Kaffenberger

RSpec Given/When/Then with Symbols — An interesting, alternative way to structure a BDD feature in RSpec. I think I prefer the underscores but YMMV.

Caius Durling

Looking Inside a Ruby Gem — Piotr decomposes a .gem file which turns out to just be a collection of gzipped and tarred files, only some of which are the code.

Piotr Murach

eBook: The Most Important Events to Monitor in Your Postgres Logs — In this eBook, you will learn about the Top 6 Postgres log events for monitoring query performance and preventing downtime.

pganalyze sponsor

Passing Rails Controller Params to SidekiqActionController::Parameters can give Sidekiq issues.

Prathamesh Sonpatki

Catchup Subscriptions with Rails Event Store

Miroslaw Praglowski

Logic-less Ruby Templates with Mustache

David Santangelo

▶  Discussing Ruby for Good with Sean Marcia — Sean talks about founding Ruby For Good (an event about philanthropic Ruby development) and some of the projects it has been responsible for creating.

Ruby Rogues podcast

???? Code and Tools

Impressionist 2.0: A Plugin to Log Impressions in Rails Apps — Impressionist tracks page views and impressions. v2.0 has just dropped but they’re also are looking for new maintainers, so contact them if you want to get involved.

Charlotte Ruby Group

acli 0.3: A Command Line Client for Action Cable — Interesting on two fronts.. first, because it’s an mruby app, and we don’t see many of those, and second, because it lets you play with Action Cable channels in any easier way.

Vladimir Dementyev

Undercover: A Tool to Stop You Shipping Untested Code — It’s like RuboCop but for code coverage rather than code style.

Jan Grodowski

How to Monitor Your Host Metrics Automatically

AppSignal sponsor

Bridgetown: A Modern Ruby (JAMstack) Web Framework — Bridgetown is a new Ruby-based static-site generator based on a fork of Jekyll. It supports plugins and Webpack, so you can use your front-end framework of choice.

Bridgetown

net-ssh 6.0: A Pure Ruby Implementation of the SSH2 Client Protocol — Yes, you can write programs that invoke and interact with processes on remote servers, via SSH2, all in Ruby.

Buck, Fazekas, et al.




in

An interview with Ruby ETL expert Thibaut Barrère

#499 — April 30, 2020

Read on the Web

???? Occasionally we run interviews in Ruby Weekly and we're back with another one.. with long time Rubyist and Kiba maintainer, Thibaut Barrère. Be sure to check out the bottom of this issue to read it, especially if you ever run ETL jobs with Ruby! ????

Ruby Weekly

▶  Let's Build a Twitter Clone in 10 Minutes with Rails, CableReady, and StimulusReflex — You know that cloning Twitter in 10 minutes is impossible, but what about the core mechanism of the idea? After a slow first minute, this video does a pretty good job of showing off some techniques you might not have used before.

Nate Hopkins

discuss.rubyonrails: The Rails Project Discussion Forum — Basically a Web version of the Rails mailing lists and a worthwhile place to head if you want to suggest features, ask questions, etc.

Ruby on Rails Discussions

Easy Rails Deployments — Deploy your Ruby, Rails, Sinatra, and Rack application to any cloud or server. Cloud 66 offers a scalable Heroku alternative that doesn't lock you in. Try it free and get extra $66 free credits with the code: 'Ruby-Weekly'.

Cloud 66 sponsor

sequel-activerecord-adapter: Allows Sequel to Reuse an ActiveRecord Connection — If you want to use more Sequel or migrate in that direction, this makes it easy.

Janko Marohnić

Ruby Adds Experimental Support for 'End-Less' Method Definitions — We’ve touched on this in a previous issue, but if you fancy a proper blog post with examples, this is more accessible than digging through feature tracker discussions. I’m not a fan of this syntax myself yet, but Prateek does a good job of selling it.

Prateek Choudhary

CableReady: Trigger Client-Side DOM Changes from Server-Side Ruby — If you skipped the video above because it’s a video (and I know many of you do ????) CableReady is still worth checking out. It aims to “complete the ActionCable story” by providing a way to directly interact with clients over ActionCable WebSockets. The docs will help you get the idea.

Hopsoft

???? Jobs

Find a Job Through Vettery — Vettery specializes in tech roles and is completely free for job seekers. Create a profile to get started.

Vettery

Security Engineer (Remote) — Are you an engineer with experience in Rails and/or Go? Join our team and help secure our apps and cloud infrastructure.

Shogun

ℹ️ Interested in running a job listing in Ruby Weekly? There's more info here.

???? Articles & Tutorials

A Practical Use for PStorePStore is one of the older parts of Ruby’s stdlib and lets you persist (and restore) a Ruby hash to disk. I haven’t seen it in Ruby code for years but Arkency have found a neat, modern use case.

Paweł Pacana

▶  Drag and Drop Sortable Lists with Rails and Stimulus JS — How to wire up drag and drop lists with a Rails app using Sortable, Stimulus and acts_as_list.

Go Rails

Setting Up Multi-Factor Authentication for RubyGems.org — If you’re a registered user of the official Ruby Gems repository, you should have had an email this week about securing your account using 2FA. This is just my own reminder that this is a good idea especially if you publicly publish widely used gems! :-)

Rubygems.org

Let’s Explore Big-O Notation With Ruby ???? — Learn to use Big-O to look at an algorithm and easily discern its efficiency, without having to run a profiling tool.

Honeybadger sponsor

▶  Discussing Docker and Kubernetes with Kelsey Hightower — A worthwhile show to listen to if Kubernetes and Docker intimidate you but you want to know a little more. Kelsey is good at breaking these things down into understandable pieces.

Rails with Jason Podcast podcast

Rails System Tests in Docker — We’re seeing an uptick in articles about system tests in Rails. Here’s how to integrate them into your development Docker setup.

Hint.io

The Difference Between System Specs and Feature Specs — If you’ve felt the difference between RSpec’s “feature specs” and “system specs” is quite subtle, this explanation will help.

Jason Swett

Why Rubyists Should Consider Learning Go — If you want compilation and a type system, Crystal is probably a better fit for Rubyists, but Go is undoubtedly a neat language and ecosystem (and if you do end up in the Go world, check out our Go weekly! ????)

Ayooluwa Isaiah

Building a Ruby CLI with Thor

Daniel Gómez

???? Code and Tools

git curate: Peruse and Delete git Branches Ergonomically — Got a repo cluttered with branches here and there? git curate aims to cure the pains of getting those branches back under control.

Matt Harvey

MessageBus: A Reliable and Robust Messaging Bus for Ruby and Rack

Sam Saffron

Are You Spending Too Much on Heroku?

Rails Autoscale sponsor

bootstrap_form: A Rails Form Builder for Bootstrap v4-Style Forms

Bootstrap Ruby

ActiveModelAttributes: The Active Record Attributes API, but for Active Model — Brings some of the goodies of the Rails 5 Active Record attributes API to ActiveModel too. 1.6.0 just dropped.

Karol Galanciak

???? A Q&A with…
Thibaut Barrère
Creator of Kiba ETL

Thibaut Barrère is a long-time Rubyist and data engineer who built and maintains the popular Extract, Transform, and Load (ETL) framework Kiba. We asked him some questions about his work:

What inspired you to create Kiba?

A lot of my work since ~2005 has been focused on data integration (making systems speak together), data aggregation etc. I sometimes used GUI-based tools like Microsoft SSIS which, while powerful, are quite far from the coding experience.

I was already using Ruby at that time, and was happy to discover activewarehouse-etl (maintained by Anthony Eden, who runs DNSimple now) providing a Ruby DSL to declare data pipelines. I used it for a while with very good success to implement data extractions and business intelligence ETL, and ultimately took over the maintenance.

In the long run, though (as I explained in a recent Paris.rb talk), the balance between the the cost of OSS maintenance and the usability for my billable and non-billable use-cases proved to be not good enough, which made me decide to stop the maintenance, sadly.

I still wanted to use Ruby to write data pipelines, but I needed to cover more use-cases and reduce the OSS maintenance burden at the same time. This ultimately led me to write and share (in 2015) Kiba ETL, a focused DSL for declarative processing, matching those criterias.

Do you find yourself adding more features while in quarantine?

Before the quarantine, I directed my Kiba bandwith and focus on finalizing Kiba v3 and rewriting the documentation from scratch, to properly encourage best practices I’ve been discovering. I also created experimental branches for Ruby 2.7/2.8 keywords.

During the quarantine, I've reduced client work and OSS work too, to focus on shipping Kiba Pro v2 (which I’ll announce shortly officially). I’ve extracted and generalized (from real-life projects) very useful components, such as a “batch SQL lookup” (useful when replacing relationships keys during data migrations and datawarehouses sync code, in batch rather than row by row), a “file lock” to ensure a single job runs at once, and a “parallel transform” to achieve easy concurrency for things like HTTP queries.

What's the wildest ETL that you've encountered?

Getting the data out of a system which is actively not acting in that direction is always a bit wild.. One can see all types of fancy stuff on the field. For instance, it is not uncommon to have an ETL process start a headless browser, jump through pages, just to get to the CSV/PDF/Excel file that you will then use as your data source!

You can also end up having to figure out ways to read or write very old file formats at times. Recently I wrote a Kiba component to generate a COBOL delimited file, for instance. In large companies, a very widely used I/O is good old SFTP, far away from modern APIs and formats.

Can you tell us how to say your last name? ????

I had to deploy a page to my blog to answer that question properly ????. You’ll find out how to say my name here.

Merci Thibaut!

You can read some of Thibaut's posts on his blog and find out more about Kiba ETL here.




in

Can you build Node add-ons in Rust? Yes.

#336 — April 30, 2020

Read on the Web

Be sure to check out the Tools and Libraries section today as there have been quite a lot of (minor) releases.. from MIDI parsing and JPEG decoding to generating TypeScript types from a Postgres database.. maybe there's something for you ????

Node Weekly

Middy 1.0: A Node Middleware Framework for AWS Lambda — Middy’s aim is to make writing serverless functions (hosted on AWS Lambda) easier by providing a familiar middleware abstraction to Node developers. The example in this post shows off the main benefit.

Luciano Mammino

Rust and Node.js: A Match Made in Heaven? — This is technical stuff but using other languages, such as Rust, for building add-ons for Node is an interesting area.

Anshul Goyal

Faster CI/CD for All Your Software Projects Using Buildkite — See how Shopify scaled from 300 to 1800 engineers while keeping their build times under 5 minutes.

Buildkite sponsor

Editly: Slick, Declarative Command Line Video Editing — I’ve long wondered why there isn’t a good way to “code” video editing at the command line other than wrangling with arcane ffmpeg options. Well.. this uses ffmpeg, but it handles a lot of the wrangling for you.

Mikael Finstad

Node v14.1.0 (Current) ReleasedLast week we featured the release of Node 14.0 and 14.1 is already with us. Principally bug fixes, plus an update to the OpenSSL dependency.

Bethany Nicolle Griggs

???? Jobs

Backend Developer (Skien, Norway) — We are looking for a full-stack dev with a solid track record to help us adapt to tomorrow's security requirements.

OKAY

Find a Job Through Vettery — Vettery specializes in tech roles and is completely free for job seekers. Create a profile to get started.

Vettery

ℹ️ If you're interested in running a job listing in this newsletter, there's more info here.

???? Articles & Tutorials

Four Tools for Web Scraping in Node — A walk through of a few different libraries (for scraping and parsing data directly from websites) to see how they work and how they compare to each other.

Sam Agnew

Six Platforms for Hosting a Node App in 2020 — Of course, you can run a Node app pretty much anywhere there’s a server, but some platforms make it easier than others. These all have free tiers too. Glitch, Now.sh (now Vercel) and Heroku are particular favorites of ours at Cooperpress.

Amit Bendor

Getting Started with NuxtJS — Learn how to create Vue.js-powered server-side rendered apps with NuxtJS including configuring an app and deploying it on Heroku.

Timi Omoyeni

The Node.js Security Handbook — Improve the security of your Node.js app with the Node.js security handbook made for developers.

Sqreen sponsor

A Collection of Challenging TypeScript Exercises“The goal: Let everyone play with many different TypeScript features and get an overview of TypeScript capabilities and principles.”

Marat Dulin

Exploring Node.js Internals — It’s reasonably elementary but Aleem Isiaka explains how the internals of Node.js interact with one another on a simple task such as creating a file.

Smashing Magazine

Creating CommonJS-Based npm Packages via TypeScript

Dr. Axel Rauschmayer

Turning Vue Components Into Reusable npm Packages — Outlines how you can reuse Vue components across your projects by automating your process to bundle, test, document, and publish your components.

Sjoerd de voorhoede

???? Tools, Resources and Libraries

Node v12.16.3 (LTS) Released — OpenSSL gets an update, and warnings are no longer printed for modules that use conditional exports or package name self resolution.

Node.js

pm2 4.4 Released: The Node Production Process Manager — A very mature and widely used process manager that includes a load balancer for keeping Node apps alive forever and to reload them without downtime. v4.4 improves the Node 14 compatibility.

Alexandre Strzelewicz

jpeg-js: A Pure JavaScript JPEG Encoder and Decoder — It admits it’s far slower than native alternatives but if you need a pure JavaScript JPEG encoder/decoder, this is where to go.

Eugene Ware

AppSignal Now Supports Node.js: Roadmap for the Coming Weeks

AppSignal sponsor

node-stream-zip: For Fast Reading of ZIP Archives — Reads chunk by chunk rather than all in one go so it’s memory friendly.

Dimitri Witkowski

JZZ: A MIDI Library for Node and Web Browsers — Send, receive and play MIDI messages from both Node and the browser on Linux, macOS and Windows.

Sema

Vegemite: A Pub/Sub State Manager — Inspired by Immer and Redux, full TypeScript support, and sized at only 623 bytes, which includes one dependency.

Luke Edwards

Kanel: Generate TypeScript Types from Postgres

Kristian Dupont

web-worker: Consistent Web Workers for the Browser and Node — In Node it works as a web-compatible Worker implementation atop worker_threads. In the browser it’s an alias for Worker.

Jason Miller

node-csv-parse: A CSV Parser Implementing the stream.Transform API

Adaltas




in

Node 14.2.0, plus Deno 1.0 is coming

#337 — May 7, 2020

Read on the Web

✍️ With a few of the links today, this is a good time to note we sometimes link to things we disagree with or that are controversial if they are newsworthy or of relevance to our community. Inclusion is not always endorsement but you can read any summaries we write alongside items to get our take on things ????

Node Weekly

Node v14.2.0 (Current) Released — The latest version of Node gains a new experimental way — assert.CallTracker — to track and verify function calls and the amount of times they occur. Also, require('console').Console now supports different group indentations

Node.js

Deno 1.0: What You Need to Know — Two years ago Ryan Dahl, the original creator of Node, gave a popular talk called 10 Things I Regret About Node.js where he revealed Deno, his prototype of how he'd build a better V8-based JavaScript runtime. With 1.0 due next week, Deno is poised to be a particularly exciting release and this article does a good job of cruising through the reasons why.

David Else

Enhance Node.js Performance with Datadog APM — Debug errors and bottlenecks in your code by tracing requests across web servers and services in your environment. Then correlate between distributed request traces, metrics, and logs to troubleshoot issues without switching tools or contexts. Try Datadog APM free.

Datadog APM sponsor

Deno Weekly: Our Newest Newsletter — We really like what we see from Deno (above) so far, so we're launching a new newsletter all about it! ???? Rather than keep mentioning Deno in Node Weekly, we'll be giving it its own space. Even if you're not planning to use Deno, feel free to subscribe for a while, see what happens, then unsubscribe if it doesn't suit you — the next issue will drop on 1.0's release (due next Wednesday).

Cooperpress

Controlling GUIs Automatically with Nut.js — Write Node code that clicks on things, opens apps, types, clicks buttons, etc. Works on Windows, macOS and Linux. Hit the GitHub repo to learn more or check out some examples.

Simon Hofmann

A Practical Guide to Node Buffers — You’ll often encounter Buffer objects for holding binary data in the form of a sequence of bytes during interactions with the operating system, working with files, network transfers, etc.

DigitalOcean

???? Jobs

Node.js Developer at X-Team (Remote) — Join X-Team and work on projects for companies like Riot Games, FOX, Coinbase, and more. Work from anywhere.

X-Team

Find a Job Through Vettery — Vettery specializes in tech roles and is completely free for job seekers. Create a profile to get started.

Vettery

ℹ️ If you're interested in running a job listing in this newsletter, there's more info here.

???? Articles & Opinions

How to Build a REST Service with Fastify — How to build a basic RESTful service using Fastify, a popular Node Web framework focused on performance/low overheads.

Wisdom Ekpot

▶  How to Use Node.js for Load Testing — A straightforward tour of an approach for hitting a Web site over and over from multiple child processes.

Tom Baranowicz

How to Fix ESLint Errors Upon Save in VS Code — A quick fire tip.

David Walsh

Faster CI/CD for All Your Software Projects Using Buildkite — See how Shopify scaled from 300 to 1800 engineers while keeping their build times under 5 minutes.

Buildkite sponsor

Avoiding Memory Leaks in Node: Best Practices for Performance — Covers very similar ground to another memory leak article we linked recently.

Deepu K Sasidharan

'Some thoughts on the npm acquisition..' — The creator of Hapi and an investor in npm Inc. shared his thoughts on GitHub’s acquisition of npm. I disagree with his conclusion (and his views have also caused concern on Twitter) but it’s nonetheless interesting to get views from behind the curtain.

Eran Hammer

???? Tools, Resources and Libraries

npm 6.14.5 Released — Just a couple of minor bug fixes.

The npm Blog

actions-cli: Monitor Your GitHub Actions in Real Time from the Command Line

Tommaso De Rossi

SQL Template Tag: Tagged Template Strings for Preparing SQL Statements — For use with pg and mysql, for example.

Blake Embrey

webpack-blocks: Configure webpack using Functional Feature Blocks

Andy Wermke

JavaScript Error Tracking with AppSignal v1.3.0 is Here

AppSignal sponsor

FarmHash 3.1: A Node Implementation of Google's High Performance Hash FunctionsFarmHash is a family of non-cryptographic hash functions built by Google mostly for quickly hashing strings.

Lovell Fuller

do-wrapper 4.0: A Node Wrapper for the DigitalOcean v2 API

Matthew Major




in

A CLI podcast player built in Go

#308 — April 17, 2020

Unsubscribe  :  Read on the Web

Golang Weekly

Broccoli: Using Brotli Compression to Embed Static Files in Go — There’s been talk about making static file embedding a standard part of Go, but for now you might find this project interesting. It uses the Brotli compression system to embed a virtual file system of static files in your Go executables as tightly as possible.

Aletheia

How Thanos Would Program in Go — An introduction to the Thanos Go Style Guide built for Thanos, the distributed metrics system project, not the Marvel super-villain, BTW ????

Bartek Płotka

Introducing GoLand 2020.1 — A variety of upgrades for Go Modules support, code-editing features that require little to no interaction from the user, an expanded code completion family, and more! Try free for 30 days.

GoLand sponsor

Understanding Bytes in Go by Building a TCP Protocol — There is a lot more in this long-ish tutorial than just learning about bytes. This is great if, let’s say, you are stuck at home and need a challenge. (Note: If you’ve got deja-vu, we linked this in last week’s brief non-issue.)

Ilija Eftimov

Ebiten 1.11.0 Released: The Go 2D Gamedev Library — Ebiten is one of those genuine gems of a project. Maybe use it to take part in this weekend’s Ludum Dare game jam? More Go entries would be neat..

Ebiten

Generics in Go: How They Work and How to Play With Them — Generics are a lot closer than you might think. So much so that you can try them today in a browser or compile locally.

Chris Brown

???? Jobs

Senior Software Engineer (Go) – 100% Remote (UK/EU Only) — Form3 is building the most exciting banking technology on the planet and are looking for Talented Engineers to join the team.

Form3

Golang Developer at X-Team (Remote) — Join X-Team and work on projects for companies like Riot Games, FOX, Coinbase, and more. Work from anywhere.

X-Team

Find a Job Through Vettery — Vettery specializes in tech roles and is completely free for job seekers. Create a profile to get started.

Vettery

???? Articles & Tutorials

Statically Compiling Go Programs — If you thought all/most Go binaries were static, you might be surprised to find out that some core packages use cgo code and result in dynamically linked libraries.

Martin Tournoij

How To Create Testable Go Code — Structure your code and tests to be mockable, testable, and maintainable, even if it calls external services.

Dave Wales

The Go Security Checklist — Ensure the infrastructure and the code of your Go applications are secure with the latest actionable best practices.

Sqreen sponsor

Build Your Own Neural Network in Go — A beginner’s guide to building the simplest parts of a neural network completely from scratch.

Dasaradh S K

'How I Built a Cloud Gaming System with WebRTC and Go'

Thanh Nguyen

???? Code & Tools

podcast-cli: A Podcast Player with a Terminal-Based Interface

Goulin

Godocgen: A Go Documentation Generator — Godocgen can output to multiple formats/destinations, making it easy to host as a static site. More background here.

Holloway Chew Kean Ho

3mux: An i3-inspired Terminal Multiplexer — Imagine something like tmux but easier to learn and with sensible defaults. Plus, it’s written in Go so you can tweak it as much as you like :-)

Aaron Janse

Micro 2.5: A Go Micro Services Development Framework

Micro

Beta Launch: Code Performance Profiling - Find & Fix Bottlenecks

Blackfire sponsor

Goph: A Native Go SSH Client — Supports connections using passwords, private keys, keys with passphrases, doing file uploads and downloads, etc.

Mohamed El Bahja

GeoDB: A Persistent Geospatial Database with Geofencing and Google Maps Support — Built using Badger gRPC and the Google Maps API. Track the geolocation of objects across boundaries or in relation to other objects.

Coleman Word

oneinfra: A 'Kubernetes as a Service' Platform — Provide or consume Kubernetes clusters at scale, on any platform or service provider.

oneinfra

Gocorona: Track COVID-19 Statistics From Your Terminal — A short and sweet demonstration of what you can throw together quickly using termui, a customizable Go-powered terminal dashboard and widget library.

Ayooluwa Isaiah




in

An insightful interview with Go's Rob Pike

#310 — May 1, 2020

Unsubscribe  :  Read on the Web

Golang Weekly

An Interview with Go's Rob Pike — Go’s co-creator answers some big picture questions about Go’s status, history, and future. “Go has indeed become the language of cloud infrastructure,” says Rob.

Evrone

???? What's Coming in Go 1.15 — This presentation covers all the major sections: tooling, performance, API changes, and the Big Ones, like the aforementioned smaller binaries. Fingers crossed for a final release in August.

Daniel Martí slidedeck

Troubleshoot Golang App Issues with End-To-End Distributed Tracing — Trace requests across service boundaries to optimize bottlenecks by drilling into individual traces end-to-end with flame graphs. Correlate Golang traces with related logs and metrics for fast troubleshooting. Enhance performance with a free Datadog APM trial.

Datadog APM sponsor

Debugging Go Programs using Delve — The recent Go community survey showed that most Go developers use text-based logging (e.g. with fmt.Print()) to debug, but if you want to step things up a notch, this is a gentle intro to Delve.

Naveen Ramanathan

My Journey Optimizing The Go Compiler — Assel explains how a simple task evolved into a legitimate compiler optimization (aimed at 1.15) and proves we should all have a curious mind.

Assel Meher

The 'Ultimate' Go Study Guide — A large repository of code examples with comments and notes from Hoanh’s attempt at learning the language. If you pick up concepts well from straightforward examples, this is worth a look.

Hoanh An

???? Jobs

Software Engineer at HiPeople (Remote/Berlin) — Fast-moving startup (backed by top tier VCs) shaping the future of modern recruiting is looking for engineers who love working with Go.

HiPeople

Find a Job Through Vettery — Vettery specializes in tech roles and is completely free for job seekers. Create a profile to get started.

Vettery

???? Articles & Tutorials

Making a Multiplayer Game with Go and gRPC — Started as a (somewhat ambitious) project to learn Go, Sam walks us through the algorithms, design decisions, mistakes, and where Go helped and hurt the game.

Samuel Mortenson

Documenting a Go GitHub Repo — Or, “How to Keep the README in Your GitHub Repo in Sync with Your Go Doc.”

Eyal Posener

The 5 Crucial PDF & Office Features For Corporate Apps in Pure Go — UniDoc develops pure Go libraries for managing PDF and Office files since 2016. Here are the features developers use the most.

UniDoc sponsor

▶  Discussing Building Immediate Mode GUIs in Go — Elias Naur, creator of Gio, joins the popular Go podcast to discuss building GUI apps with Go, the pros and cons of immediate vs retained mode and examples of each.

Go Time Podcast podcast

The Creation of a Realtime Patient Monitoring System with Go and Vue in 3 Days — This is the Go content I am here for. Connecting with monitoring devices and leveraging Go’s strengths to create a helpful, distributed application. Great work.

Kasun Vithanage

Add It Up: Azure’s Go Problem — Here’s one takeaway from the Go Developer Survey. Of the major clouds, Azure is the one Go developers seem least enamored by.

Lawrence E Hecht

Why You Should Generally Be using the Latest Version of Go — No surprising arguments here.

Chris Siebenmann

???? Code & Tools

XLSX: A Library for Reading and Writing XLSX (Excel) Files — Got spreadsheets? Want to make spreadsheets? There’s a lot you can do with them here.

Geoffrey J. Teale

SQLBoiler: Generate a Go ORM Tailored to Your Database Schema — A long standing library that has now switched to modules.

Volatile Technologies Inc.

Decimal: Arbitrary-Precision Fixed-Point Decimal Numbers for Go — The library laments that it can only support decimal numbers with up to 2^38 digits after the decimal point so take care ????

Spring Engineering

Beta Launch: Code Performance Profiling - Find & Fix Bottlenecks

Blackfire sponsor

Redigo: A Go Client for Redis — In related news, Redis 6.0 has just been released.

Gary Burd

ntp: Facebook's NTP Libraries — NTP stands for “Network Time Protocol”, if you were wondering. Basically, clock synchronization.

Facebook Incubator

grobotstxt: A Native Go Port of Google's Robots.txt Parser and Matcher Library — Now you can crawl your own site, just like Google does.

Jim Smart

A Compiler for a Small Custom Language Into x86-64 Assembly — One of those ‘labor of love’ type projects that you might enjoy poking around in. You won’t use this project directly, but you might be intrigued how to create a similar compiler for your own thing.

Maurice Tollmien

MIDAS: Microcluster-Based Detector of Anomalies in Edge Streams — A Go reimplementation of this C++ version.

Steve Tan

Liftbridge 1.0: Lightweight, Fault-Tolerant Message Streams — A server that implements a durable, replicated message log for the NATS messaging system.

Liftbridge




in

Ellie and Nele: From she to he - and back to she again

When two trans men fell in love they began to have second thoughts about their gender.




in

'How do I convince the Home Office I'm a lesbian?'

More than 1,500 people claim asylum in the UK each year, claiming that they are persecuted for being gay. But it's not an easy thing to prove.




in

Blasian love: The day we introduced our black and Asian families

Blasian - black and Asian - couples now exist in South Africa... but they don't always have an easy time.




in

'My search for the boy in a child abuse video'

Lucy Proctor was horrified when she was WhatsApped a sex abuse video. And she wanted to find out if the boy was safe.




in

Coronavirus: The month everything changed

In the space of a month, the United Kingdom has transformed beyond recognition.




in

‘My toy walrus waited 25 years in the Arctic’

Julia spent 25 years dreaming of her first home. Eventually she returned - and found a long-lost toy.




in

The rapper's track that sparked a wave of killings

Tensions have long existed between gangs in Tottenham and Wood Green - for 10 weeks in 2018 they boiled over.




in

The volunteer army helping self-isolating neighbours

The coronavirus outbreak has left many feeling trapped indoors, but for some help may not be far away.




in

Coronavirus: Here's how you can stop bad information from going viral

Experts are calling on the public to practise ‘information hygiene’ to help stop the spread of falsehoods online.




in

Birth in a pandemic: 'You are stronger than you think'

Coronavirus is throwing many birth plans up in the air and leading some health trusts to increase home births.




in

Coronavirus: The grandad who became a TikTok star without realising it

Joe Allington was persuaded to dance on TikTok for the first time in January. Now he's got 1.5 million followers.




in

Extreme piercing: A festival of self-inflicted pain

Since time immemorial, a village in West Bengal has paid tribute to Shiva by self-inflicting pain. This year coronavirus put a stop to it.




in

Coronavirus: 'Depression feels like my cat is sitting on my chest'

Two young people describe how the coronavirus pandemic and the lockdown have affected their mental health.




in

Why being gay in Russia is about "love and passion"

The secret moment between two gay Russian lovers that defied haters.




in

The man who grew his own Amazon rainforest

Fighting back against destruction in the Amazon: how one man grew a rainforest of his own.




in

Coronavirus: Love and dating in lockdown

How dating and love continue for three couples during the coronavirus lockdown.




in

Coronavirus: DIY hair shaving and beauty treatments

As hair dye and clippers become the next thing on the stockpile list - we look at how people are managing their hair and beauty.




in

From patient to healer: How this woman is saving lives

Women who have overcome depression are running therapy sessions to help others




in

Dirty streaming: The internet's big secret

Figures suggest that IT now generates as much CO2 as flying, with some arguing it's nearly double.




in

The Valentine's Day snake puzzle

Why were 29 snakes left in pillowcases in a Sunderland dustbin the day before and the day after Valentine's Day?




in

Ramadan and Coronavirus: Breaking my fast on Zoom

How fasting in lockdown and isolation has changed Ramadan for young Muslims this year.




in

Coronavirus: 'I'm being bombarded by gambling ads'

Gambling companies have halted TV and radio ads during lockdown - but not online ads.




in

Coronavirus: A toast to my cancelled wedding

Today was going to be my big day until Covid-19 intervened. But that won't stop me delivering my speech.




in

Coronavirus career pivots: 'I now work in a supermarket'

An actress and a commercial sales leader talk about making the switch to working in a supermarket.