Posts on xmatthias blog

Debugging a pytest segfault

Fri, 08 Dec 2023 19:00:00 +0000

Intro

A recent update of pytorch (update from 2.0.1 to 2.1.1) broke the windows pytest CI runs of the project I maintain, Freqtrade. Pytest itself showed nothing helpful beyond a few warnings. All tests were passing, yet the CI step was failing.

That’s however a great opportunity to learn how to approach such issues.

Disclaimer This Post will be quite technical and should hence be considered advanced content. I’ll assume basic understanding of pytest, too.

Reproducing the problem

The initial failure happened on windows - but setting the same “random state” to all systems quickly showed that also MacOS / linux behave identically with a specific sequence.

The linux output was slightly more helpful by providing a Segmentation fault (core dumped) output. Yet pytest did not provide any information on the test causing this, nor any further hint about where of what might be happening.

To start debugging this in github actions, i added the following action step after each of the test steps.

    - name: Setup tmate session
      if: failure()
      uses: mxschmitt/action-tmate@v3

This allows me to ssh into each of the Actions, and perform an analysis on the affected system. While time is limited to ~6h (the maximum execution time on github Actions) - that’s plenty of time to run a few test.

I could rather quickly confirm that the problem happens only with torch==2.1.1. torch==2.0.1 did not show the error, and tests were working perfectly fine.

A pytest issue from 2021 showed that an older pytorch version had similar problems. While it didn’t really offer a solution, it did offer the following command to get to the traceback of the segfault

python3 -X faulthandler -m pytest -p no:faulthandler

This disables the pytest error-hander (faulthandler), and reverts to the faulthandler built into python.

python3 -X faulthandler -m pytest --random-order-seed 208417 -p no:faulthandler

(the actual random state is not relevant, but this seed happened to cause the problem initially).

This allowed me to pinpoint the problem to some part of the cleanup code. What’s odd here is the fact that the actual error seems to come from the logging module.

The code itself (__del__() calling close() - which adds log messages) was at this point multiple years old, and never caused any problem (neither during tests, neither during regular executions).

At this point, i was still testing with the full test suite - which took around 5 minutes before exiting. The error also happened about 5-10 seconds after the last output from pytest (which might be a tell-tale sign). Not a great situation to be debugging this.

Reducing the test size

By this point, i was able to reproduce the same issue locally - which simplified the further steps.

However, the failure did not happen without changing the test execution sequence to something random - hence I needed some way to keep the problematic test sequence, while still reducing the number of tests that run.

Pytest allows us to collect the tests that will be running. Hence we can pipe this output into a file, which we’ll slowly reduce until only a few tests are left.

pytest --random-order-seed 208417  --collect-only -q > base_torch_tests.txt

We need to open the file and remove the final 2 lines (it’s important that there is no empty line at the end, otherwise the full test-suite will run again after the tests in the file).

We can then use a quite handy xargs script to run the tests in the file.

xargs -d '\n' pytest -v < torch_tests_base.txt

for MacOS, you’ll need a slightly different command, as xargs on Mac doesn’t have a -d option.

cat torch_tests_base.txt | tr '\n' '\0' | xargs -0 pytest -v

Next, we move through the following steps until only 2 tests are left

modify the file, removing tests
rerun above command
repeat

The method in this section is taken from Anthony Sottile’s [Youtube Video](reducing the amount of tests), which explains the whole process in greater detail.

The problem

Once only 2 tests were left, i tried to get behind what might be causing a failure out of the logging module.

The odd thing about the tests: The first test must somehow import torch (it didn’t even use torch, just imported a file which happened to import torch) The 2nd test was completely unrelated to torch - but both touched the __del__() method we saw above.

Removing the log line, no segfault (so it’s related to the logging module).

As we only import torch - the next step is to analyze what torch is actually doing when it’s imported (spoiler: a lot).

The offending code is in the torch __init__.py file - which “initializes logging”.

I tried to dig a bit deeper into what this is actually doing that’s causing logging to fail at a later point, but I’m still at a loss (Please get in touch if you happen to know the reason).

The “quick and dirty” fix for this problem is to mock the torch init_logs function during test executions. By marking the pytest fixture as autouse=True, it’s automatically used for all tests.

@pytest.fixture(autouse=True)
def patch_torch_initlogs(mocker) -> None:
    mocker.patch("torch._logging._init_logs")

While that’s not an ideal solution - it’s a test-problem only, hence justifies a test-only solution.

The Final fix looked slightly different due to another problem on MacOS. Tests involving Torch on this platform were already disabled - so we now mock the whole module to avoid failures due to importing this.

@pytest.fixture(autouse=True)
def patch_torch_initlogs(mocker) -> None:

    if is_mac():
        # Mock torch import completely
        import sys
        import types

        module_name = 'torch'
        mocked_module = types.ModuleType(module_name)
        sys.modules[module_name] = mocked_module
    else:
        mocker.patch("torch._logging._init_logs")

Wrapping up

This whole investigation took quite a few hours out of my week, but was well worth my time. I learned quite a lot (how to debug in github actions, how to debug segfaults that pytest “hides”, how badly some libraries treat the python environment).

Migrating from ghost to hugo in 2023

Sat, 23 Sep 2023 18:00:00 +0000

Reasons

I’ve been running this blog on a self-hosted ghost platform ever since the very first post. Over time, ghost served me well - and this is not a decision against ghost, but for a simplified site setup.

I’ve been eyeing at hugo for some time now. As it builds a static webpage, there are tons of hosting options (free and paid). To me, this provides the benefit of being able to decomission the server I used to run the blog on.

In this blogpost, I’ll outline the steps I took to migrate the existing content to the new blog.

Initial setup

To get started, I used ghostToHugo - a fork of the original ghostToHugo project, to convert all posts in Ghost to Hugo templates.

This is as easy as creating an export from the ghost admin panel (Settings -> labs -> "Export your content") and downloading the resulting .json file. We then run ghostToHugo -p ~/mysite export.json - which creates a hugo project for us.

Next, I took an existing simple theme and configured this for my new hugo project

git clone https://github.com/hugo-sid/hugo-blog-awesome.git themes/hugo-blog-awesome

We then add this theme to our hugo.toml file as theme = "hugo-blog-awesome".

At this point, we already have an initial page up and running, which we can start with hugo server.

Update links and assets

It’s worth pointing out that links between posts won’t work at this point, nor will images render correctly.

Inspecting one of the posts, it becomes immediately clear why - ghost’s export added a __GHOST_URL__ in front of all links. [link](__GHOST_URL__/duplicity_remote/)

We can fix this pretty easily by removing all occurances of __GHOST_URL__/ (for the above case, this leaves [link](duplicity_remote/)).

Images have a similar behavior - though slightly more complicated, as ghost stores them as follows: ![onedrive_oauth1](__GHOST_URL__/content/images/2023/08/some_image.png)

As I like to keep the post and corresponding image together, I used the following approach (shown on the example of a fictionary post hello-world.md):

Create a directory next to the hello-world.md file.
Move the file into the directory.
Move corresponding image files into the directory (i happened to have the original files still in a backup repository).
Replace image sources.

mkdir content/post/hello-world/
mv content/post/hello-world.md content/post/hello-world/index.md

mv ../imagesource/hello-world.jpg content/post/hello-world/

We can now replace the link to be a relative link: ![onedrive_oauth1](__GHOST_URL__/content/images/2017/10/some_image.png) -> ![onedrive_oauth1](some_image.png).

We obviously have to repeat this for every blogpost with images. The resulting structure is a mix of directories (posts with images) and standalone posts (posts without images).

Update Post header

Out of the conversion, all posts had a header as follows:

+++
date = 2023-08-03T11:30:00Z
description = ""
draft = false
slug = "hello-world"
title = "hello world"
summary = "Hello world blog post"
tags = ["personal"]
+++

While this is a good first step - it’s not ideal. The description was empty in all cases - however many posts had a summary.

A quick look at the Lighthouse benchmark revealed that the posts were missing a “description” field (your experience may vary depending on the selected theme).

As a quick fix, I’ve simply copied the content of “summary” to “description” - as that seemed to be fitting in most instances.

Final heading:

+++
date = 2023-08-03T11:30:00Z
description = "Hello world blog post"
draft = false
slug = "hello-world"
title = "hello world"
summary = "Hello world blog post"
tags = ["personal"]
+++

URL changes

My existing ghost blog was using url’s as follows: https://blog.xmatthias.com/hello-world/. Hugo by default inserts a /post in the middle, so the url becomes https://blog.xmatthias.com/post/hello-world/. That’s not a problem, but requires some attention, which means we have to

Add redirects - ideally as 301 (hugo-style redirects will hurt SEO).
Migrate url’s on a comment platform (in my case, disqus).

Redirects

As i’m going to deploy my hugo page on cloudflare pages, I can use the redirects feature - which is as simple as a _redirects file in the static directory (/static/_redirects)

I have the following rules.

/rss/ /post/index.xml 200
/rss /rss/ 301
# Redirect from old post addresses
/hello-world /post/hello-world/ 301
/hello-world/ /post/hello-world/ 301
# ... (continued for all posts)

The first 2 lines are there to ensure the RSS url remains the same, available under blog.xmatthias.com/rss/.

All posts will receive a permanent redirect (301) - which will allow SEO to continue to work, as the new page is treated identically to the prior page.

Disqus / comment system

For disqus, there’s a few “migration tools”. Right after putting the new page live, i verified that all redirects work - and then tried to use the “Redirect Crawler”. Unfortunately, it didn’t yield the expected results - which led me to the URL mapper.

This takes a plain csv file with from,to format - so for our sample post, it looked as follows:

https://blog.xmatthias.com/hello-world/,https://blog.xmatthias.com/post/hello-world/

This format is very similar to the one used in _redirects - and required only some changes. Saving and uploading the file gave me a confirmation box (so i could check my redirects).

After confirming that this was working, the comments started to imediately pop up on the new page.

Closing

In total (including learning hugo, and the most timeconsuming task, picking a theme), this took me about 3-4 hours over the course of a few days.

This was sufficient for me to put the blog out into the wild - though this wasn’t all - and after changing the blog engine, I did some further updates (Theme adjustments, adding security headers). This will however be a topic for another blog post.

duplicity - Backblaze

Sat, 26 May 2018 19:00:00 +0000

This is one post in a series of tutorials on Duplicity. If you are just getting started with duplicity, I would recommend to head over to the first and second part to get a basic overview of duplicity and get GPG keys setup properly (this will not be covered in this post, but is required for the tutorial to work). You should also have a look at the duplicity overview post, which contains an overview of this series. The basic configuration from this post will be used as basis in this post to configure duplicity-backup.sh.

Backblaze

To start with, we will need an account at Backblaze. Once we’re setup (verified the account via phone) and have configured 2FA (you do use 2FA, don’t you?) we can get started. We’ll get 10G Storage for free, free upload and 1Gb free download, just for signing up. This will be enough to get started using backups, and evaluate if this is the right service.

As Backblaze offers a very good tutorial on how to get duplicity setup under Linux, I’ll not repeat these steps here.

Rather, we’ll jump right in and include Backblaze into the duplicity-backup.sh script, we used previously.

duplicity-backup.sh

The following will assume that Backblaze has been setup and tested successfully using the Backblaze tutorial.

Should you have problems with the above Tutorial and getting setup let me know in the comments down below and I’ll update this post to include these steps.

Single site Backup

To backup to Backblaze using the duplicity-backup.sh script we used previously, change DEST to the following to the script:

# DEST="b2://<accountid>:<Application key>@<bucket name>"
DEST="b2://431b5d5zd3ac:dad74350dabd8d9b47ca3bc7094c784339c8452d8f@duplicitytest"

Don’t even mind trying using the above line without inserting your own id/key - this is not my account (and a randomized key).

We now run the duplicity-backup.sh script.

./duplicity-backup.sh --config duplicity-backup.conf --backup

Please note that this method is not recommended as the backup is in only one place. Depending on the requirements, the following approach should be preferred.

Multi-site Backup

As explained in this article, we can simply add backup-locations by appending them to the configuration file.

Remember how we configured our DEST variable:

DEST='"multi:///home/matt/multiconfig.conf?mode=mirror&onfail=abort"'

This mirrors all backups to all configured sites, and fails if one is not available. Now, open the file configured above and insert the Backblaze configuration

[
 {
  "description": "Local disk test",
  "url": "file:///home/matt/backupmulti"
 },
 {
  "description": "Backblaze test",
  "url": "b2://431b5d5zd3ac:dad74350dabd8d9b47ca3bc7094c784339c8452d8f@duplicitytest"
 }
]

As I already had a backup from previous backups, we need to upload the existing backups first. This way, we won’t loose any backup-history. This can be accomplished by running the following command (please make sure to update both account-id, keys and bucket name to suit your needs)

# Insert required account-id and application key when asked
b2 authorize-account

cd <existingbackupdir>
# Sync the backup-folder to the duplicity-bucket "duplicitytest"
b2 sync . b2://duplicitytest/

This way, we keep our history as well.

Closing words

There are a a ton of of options for cloud backups. Backblaze is one of them (and for now, my backup provider of choice). The speed is good (something I complained about in my Hubic Tutorial). Also the security seems good, and people I trust security-wise like Troy Hunt trust Backblaze with their data.

In the end, it does not matter all that much which provider is used, as long as the backup is executed regularly, tested and implemented following the “3-2-1” rule:

3 total copies of the data - on 2 different mediums, at least 1 off site.

Full disclosure

Title photo credit: ChrisDag via photopin (license)

Contributing to Open Source

Sat, 14 Apr 2018 19:00:00 +0000

I’ve been very quiet on this blog recently, mainly because I was trying to get started contributing to Open Source projects. I wanted to start giving back for a very long time, but never got around to actually start contributing regularly.

This all got started by Troy Hunt introducing Pwned Passwords v2 back in February. The same night, right after reading Troy’s article, I quickly spun up a python project leveraging the K-Anonymity method and checking for compromised passwords offline. While refactoring the project to upload it to pypi, I introduced a nasty bug causing the script to stop functioning. A few days later, I found a Pull Request in my repository, fixing the bug and repairing what I had broken. The simplicity of doing this amazed me - I merged the PR, released a new version and decided that I should finally find a project I can contribute to myself.

A few days later, I stumbled upon a project to which I since contribute regularly. It’s only been a few months, but I feel that I’m going to keep working on this project for a while - as I really love the code-base and direction the Project is moving to. I do link to the project below - but I don’t think it’s important which project it is, as this article is about contributing to Open Source, not about which project i picked.

How to get started

Picking a project

In order to get started, you needed a project to contribute to. My main selection criteria are below:

a small to medium size project, which already has multiple contributors (only one contributor might mean that he is not open to contributions), but not too many (huge projects mostly have also a huge code-base.
the code base shouldn’t be too big, so getting into it does not take weeks or months.
It should be in a language I am familiar with and actively use (this way, I can get the most out of it for myself, as it’s both practicing what I know, and learning new stuff as other developers will tackle problems differently).
I should be using the project myself either now or in the future, otherwise I’ll loose interest soon (my goal was/is to get actively involved longer term - not do one-off contributions).

I was not actively searching for a project, but stumbled upon a very interesting project, which happens to be freqtrade - a Trading Bot written in Python, with a few core members, a good community contributing and bringing the project forward, doing very good code reviews on pull requests (which I feel is important, as it helps bring my own coding to the next level) and a good test-suite.

What is in it for me

While working in Open Source does require some spare time, I feel that the time is a good investment. It allows me do what I love, while also collaborating with others doing what they love. This project was actually a very fortunate pick, as it brings together programmers with traders, which happen to understand the market, so I was able to get some different views on the same topic, which should help me seeing the business case for the applications I work on in my Job.

Having my code reviewed

By far the best thing is to have your code extensively reviewed and commented on. This exposure may seem intimidating at first, but its one of the best things in open source. Someone is thinking about what I wrote, trying to understand it, and may help get it corrected - or show a better way to accomplish the same thing.

Doing code reviews

I also helped in reviewing code myself, which thought me a lot by understanding how others tackle a problem, discussing the solution in case of doubt and helping others by providing a different view on a problem. Even if you don’t comment on a PR, still read through the code, try to understand the intention of the changes, it’ll help in understanding the project (and the direction the project is moving towards).

Patience

Contributing to Open Source will require patience. I read this before, but it’s absolutely true, so when I did my first contribution, I didn’t even expect it to be merged (it was merged, after doing a few modifications which came out from the Code Review). After submitting your first Pull Request, you will need patience. Core contributers may not be around immediately to review your code. It may take days, weeks or months depending on the size of the project until your PR is (eventually) merged or reviewed - and there is nothing you can do about this.

Testing

While working on this Project, I got exposed to pytest for the first time. I had heard about it a few times already, but never got a chance to actually use it. This project gave me the chance to do so, with a good base to start with (reading the code already in the repository). It does not matter which test-framework is used in the end - but having good tests is very important to not introduce regressions.

Wrapping up

I will definitely continue to contribute to Open Source Projects. Time will not always allow me to contribute at the same pace, but that’s fine. I am not forced to do any of this, but I still try to help because I like doing it, and I can learn new things by contributing.

If you’re a developer, trying to learn new things every day, you should try to contribute to open-source projects as well (if you’re not doing that already). Many projects have a good community (if they don’t, consider finding an alternative project) and there is always something new to learn (or someone to be taught something new). Learning new things by doing Courses or reading articles is important too, but it’s equally (if not more) important to apply the knowledge - that’s what will benefit you the most.

duplicity - multi-site backup

Sat, 17 Feb 2018 19:00:00 +0000

This is one post in a series of tutorials on Duplicity. If you are just getting started with duplicity, I would recommend to head over to the first and second part to get a basic overview of duplicity and get GPG keys setup properly (this will not be covered in this post, but is required for the tutorial to work). You should also have a look at the duplicity overview post, which contains a list of all posts in this series. The basic configuration from this post will be used as basis in this post to configure duplicity-backup.sh.

Multi

In this part of the duplicity series, we will try to combine several backup services into the same duplictiy backup job by using the duplicity multi backend. When setup correctly, duplicity will fulfill the 3-2-1 backup strategy all by itself, by doing one backup on the local system, and one (or more!) to remote / cloud systems.

This backend allows duplicity to use multiple backend storages at the same time, either by extending storage (mode=stripe) or by mirroring the same backup to all configured backups (mode=mirror). I will be focusing on the mirror-setup in this post, as this makes the most sense from a security/backup perspective. Stripe-setup is only useful if there is a lot of data to backup with not enough space on one cloud service (while increasing the risk of corrupted backups).

basic configuration

To get started, we use a very basic json-based configuration: Filename: /home/matt/multiconfig.conf

[
 {
  "description": "Local disk test",
  "url": "file:///home/matt/backupmulti"
 }
]

This setup will be almost identical to the initial post, where we ran the backup locally.

Now, let’s run this once:

duplicity --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB --ssh-askpass /home/matt/test/ "multi:///home/matt/multiconfig.conf"

When asked, insert your pass-phrases.

The output should look similar to mine:

duplicity --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB --ssh-askpass /home/matt/test/ "multi:///home/matt/multiconfig.conf?mode=mirror&onfail=abort"
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
GnuPG passphrase for decryption:
GnuPG passphrase for signing key:
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1518857776.80 (Sat Feb 17 09:56:16 2018)
EndTime 1518857777.17 (Sat Feb 17 09:56:17 2018)
ElapsedTime 0.37 (0.37 seconds)
SourceFiles 61
SourceFileSize 45086 (44.0 KB)
NewFiles 61
NewFileSize 45086 (44.0 KB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 61
RawDeltaSize 30 (30 bytes)
TotalDestinationSizeChange 1539 (1.50 KB)
Errors 0
-------------------------------------------------

Checking the folder we configured in the json-file above (/home/matt/backupmulti in my case) we can see that the backup was made into this folder.

Before moving to configuring a second backup site, delete the local backup (duplicity will not handle synchronizing files across multiple backup sites, so we either need to copy the existing backups to the remote backup location, or start fresh).

As we are just getting started, we can easily delete and backup again. If you already have a backup history, consider moving the existing files to all configured locations.

rm /home/matt/backupmulti/*

Now, let’s extend the json with a remote-site. I use the configuration from the initial remote backup post.

[
 {
  "description": "Local disk test",
  "url": "file:///home/matt/backupmulti"
 },
 {
   "description": "Backup to sftp site",
   "url": "sftp://matt@backupserver:1234//home/matt/offsitebackup_multi"
 }
]

We simply added a second entry to the configuration file, backing up to a remote site. Let’s run this again.

duplicity --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB --ssh-askpass /home/matt/test/ "multi:///home/matt/multiconfig.conf?mode=mirror&onfail=abort"

Notice that this time i added ?mode=mirror&onfail=abort to the end of the configuration (quoting is necessary now!). This tells duplicity to fail should one of the backups not work correctly. Combine this with one of the notification posts and we will be notified if the backup did not work. Now, let’s check both our locally configured folder, as well as the remote folder. As i deleted the backup from the first try, the folders are now identical.

duplicity-backup.sh

Next, let’s build this into our duplicity-backup script (basic configuration here).

Building on top of this configuration, we now simply change the DEST variable to the configuration used in the above command.

DEST='"multi:///home/matt/multiconfig.conf?mode=mirror&onfail=abort"'

Please notice the odd quoting - as this file will be sourced by the backup-script, we need to wrap the command we want to pass to duplicity in single-quotes - so the double-quoted string is then passed on to duplicity (with the quotes). Let’s run duplicity-backup.sh and see if our configuration does indeed work

./duplicity-backup.sh --config duplicity-backup.conf --full

Again, check both local and remote sftp folder. You should have another full backup file there, in addition to the one we had before.

We can also set this up for other backends like dropbox, simply pass in the environment variable we had to configure in the dropbox post DPBX_ACCESS_TOKEN.

[
 {
  "description": "Local disk test",
  "url": "file:///home/matt/backupmulti"
 },
 {
   "description": "Backup to dropbox",
   "url": "dpbx:///duplicitytest1",
   "env": [
     {
       "name": "DPBX_ACCESS_TOKEN",
       "value": "teBo-PD8bsUAA<redacted>"
     }
     ]
 }
]

Note that this will work for other backends like Onedrive, Hubic or Google drive as well.

Run the backup-script again.

./duplicity-backup.sh --config duplicity-backup.conf --full

Check your dropbox-folder - you should now have a new full backup in that folder.

Conclusion

In the first post of this series, I highlighted the backup-strategy everyone who cares about his data should use.

A basic rule for Backups is the 3-2-1 rule: 3 total copies of the data - on 2 different mediums, at least 1 offsite.

This part has now accomplished exactly that by having

3 copies of the data:
1. original version
2. local backup
3. remote backup
2 different mediums
1. local system
2. remote system
1 offsite
1. remote backup in the cloud (or on a different system using sftp)

We could even go further, and configure multiple (or all) Cloud services at once, and have more than 3 copies of the data - but that is overkill most of the time, but will of course depend on the importance of the data.

This post will conclude the duplicity series for now. We have now successfully setup duplicity, syncronized to various remote sites, setup notifications, and syncronized to multiple sites at once in this post.

duplicity backup - going cloud

Sat, 17 Feb 2018 19:00:00 +0000

In the upcomming parts, we will configure the duplicity-backup.sh script for different cloud services. Also, we will be using the “multi”-plugin to setup backups to multiple different servers. Personally, i would never backup to a cloud-server unencrypted - so duplicity helps by encrypting locally before uploading to the different cloud services.

I will split the posts per cloud service, so there is some order (most people will probably be interrested in only a few services). There will be one part per cloud service to keep some order. I will update this post with links to the “duplicity cloud service” posts.

In the first part, we setup our GPG keys to encrypt backups. In the second part, we looked at remote backups using sftp, scheduling and wrote a quick script for the backup. The third part covered an open source script, which made setting up different options very easy as it has an example configuration which only requires minor modifications.

Posts in this series

Duplicity backup - getting started – Setup GPG Keys for encrypted backups
Backup to remote sites – First backups to remote sites
duplicity backup script – Using duplicity-backup.sh to simplify backups with duplicity
Microsoft Onedrive – Backup to Microsoft Onedrive
hubic – Backup to hubic.com
Google Drive – Backup to Google Drive
Email Notifications – Get notified in case of problems
IFTTT notifications – Another notification method
Dropbox – Backup to Dropbox
multi – Backup to multiple sites

This post is regularily updated once new parts in this series are posted.

dupicity backup - Dropbox

Sat, 03 Feb 2018 20:00:00 +0000

This is one post in a series of tutorials on Duplicity. If you are just getting started with duplicity, I would recommend to first read the first and second part to get a basic overview of duplicity and get GPG keys setup properly (this will not be covered in this post, but is required for the tutorial to work). You should also have a look at the duplicity overview post, which contains a list of all posts in this series. The basic configuration from this post will be used as basis in this post to configure duplicity-backup.sh.

Dropbox

First of all, we obviously need a private Dropbox (Full disclosure: Referral link - non-referral link here) account. The free version currently offers 2 GB free storage, which however can be extended by inviting new people and completing some tasks. Mainly in the past, many were able to get ton of free storage from multiple different actions by Dropbox.

The dropbox-backend does require the dropbox python SDK, which we will install using the following command:

pip install dropbox==6.9.0

Note: version 6.9.0 or below is required, as version 7.1 introduced some breaking changes, which will result in error-messages saying something like TypeError: expected request_binary as binary type, got <type 'file'>.

Next, let’s log into Dropbox and go to the developer tools to create a new “Dropbox API” Application. We will use an “App folder” application here, which limits access to one folder of Dropbox instead of the whole Dropbox and name the application “duplicity_sync_xmatthias”.

Note: The Application name must be unique accross all Dropbox applications, because in theory the application could be published (which we will not do).

Click on Create App to complete the application creation. In the next screen, we can see that the application is in “development” status. All this means is that it is not published (which don’t intend to do) and only this user (and invited test-users) are allowed to use this API key.

According to the duplicty documentation, we now have 2 options - generate the API key on this site - or supply duplicity with the variables DPBX_APP_KEY and DPBX_APP_SECRET and have duplicity run through the OAuth workflow. As everyone creates his own “Application” in Dropbox, there is no benefit to this approach (this would only be viable if you share the application with someone else) - therefore we generate the Access Token in this window. Generate the Access Token as shown in the picture below.

Copy the string which will appear instead of the Button and use it as variable DPBX_ACCESS_TOKEN. (Mine is redacted as it would grant access to my Dropbox). Then, run the command we know from previous Posts to test connectivity with Dropbox.

export DPBX_ACCESS_TOKEN=teBo-PD8bsUAA<redacted>
duplicity --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB  /tmp/backupTest/ "dpbx:///duplicitytest1"

When first running this command, you may encounter an error similar to mine:

duplicity --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB  /home/matt/test/ dpbx:///duplictytest2
Exception [ApiError('87ed601063102a7aa75f9e400f9e718e', ListFolderError(u'path', LookupError(u'not_found', None)))]:
| Traceback (most recent call last):
|   File "/usr/lib/python2.7/dist-packages/duplicity/backends/dpbxbackend.py", line 76, in wrapper
|     return f(self, *args)
|   File "/usr/lib/python2.7/dist-packages/duplicity/backends/dpbxbackend.py", line 384, in _list
|     resp = self.api_client.files_list_folder(remote_dir)
|   File "/home/matt/.local/lib/python2.7/site-packages/dropbox/base.py", line 673, in files_list_folder
|     None,
|   File "/home/matt/.local/lib/python2.7/site-packages/dropbox/dropbox.py", line 272, in request
|     user_message_locale)
| ApiError: ApiError('87ed601064102f7da75f9e400f9e718e', ListFolderError(u'path', LookupError(u'not_found', None)))
Attempt 1 failed. BackendException: dpbx api error "ApiError('87ed601063102a7aa75f9e400f9e718e', ListFolderError(u'path', LookupError(u'not_found', None)))"

The duplicity documentation states that the folder needs to exist. Let’s suffice this requirement by going to our Dropbox home, navigate to “Apps/<your_app_name>” and create the folder. In my case, the url to this is https://www.dropbox.com/home/Apps/duplicity_sync_xmatthias - but your’s will differ depending on the selected name for your “application”. You can also use subfolders if multiple separate backups are required in the same account. Please note that Dropbox will sync the encrypted backup to all connected PC’s by default (unselect from “selective sync” if this is not desired).

Create the folder you’d like to use (in my case, duplicitytest1) and rerun the above command.

export DPBX_ACCESS_TOKEN=teBo-PD8bsUAA<redacted>
duplicity --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB  /tmp/backupTest/ "dpbx:///duplicitytest1"

Duplicity will now backup and upload the data.

duplicity-backup.sh

To add this to the duplicity-backup.sh script we used previously, change DEST to the following and add the access-token as DPBX_ACCESS_TOKEN to the script:

DPBX_ACCESS_TOKEN="teBo-PD8bsUAA<redacted>"
DEST="dpbx:///duplicitytest1"

The rest of the configuration remains identical to the configuration used in the duplicity-backup.sh post.

NOTE: should you receive the error BackendException: dpbx: DPBX_APP_KEY environment variable not set while running the above script, please update duplicity-backup.sh by issuing a git pull as the Dropbox-functionality required a pull-request to fully work. Also, make sure that you use the dev branch, as the PR has so far only been applied there. Previously, the access token has not been exported correctly.

We now run the duplicity-backup.sh script.

./duplicity-backup.sh --config duplicity-backup.conf --backup

A incremental backup should appear in your Dropbox folder.

You may want to now schedule the backup using the cron job as shown in the post presenting duplicity-backup.sh.

Conclusion

Everyone prefers a different cloud-storage provider. All offer different possibilities at different prices and with different advantages / disadvantages. I don’t want to judge which one others should use, and have therefore shown the setup for multiple services. Dropbox presents a good choice from a security point of view by allowing to only expose a dedicated App-folder to duplicity (we’ve seen this differently from onedrive).

In the end, it does not matter which provider is used, as long as the backup is executed regularly, tested and implemented following the “3-2-1” rule:

3 total copies of the data - on 2 different mediums, at least 1 off site.

Full disclosure

If you register with Dropbox using the link above, I will get free space added through the referral program. If you want to avoid this, please use this link or type in https://www.dropbox.com into your browser manually.

XPS 13 9360R Audio problems

Mon, 22 Jan 2018 21:00:00 +0000

Bluetooth headset

While setting up my new notebook, I discovered that my bluetooth headset would not work after suspend to RAM. It works perfectly fine when cold-booting (or restarting) - however after resume I struggled to get it to work without restarting the bluetooth service (or pulseaudio).

I did a lot of research around this area, suspecting the default configurations of either pulseaudio or bluez to be at fault, however none of the “ultimate guides for bluetooth headsets in linux” I found online did work. I also could not rule out the headset (The Dash), which has a certain age and is not without flaws.

The headset would connect, but then would immediately disconnect (or connect and not be recognized as Audio device). The log would show the following errors

Jan 18 22:30:57 xps pulseaudio[3293]: E: [pulseaudio] backend-native.c: Device doesnt exist for /org/bluez/hci0/dev_XX_XX_XX_XX_XX_XX
Jan 18 22:30:57 xps bluetoothd[2185]: Headset Voice gateway replied with an error: org.bluez.Error.InvalidArguments, Unable to handle n
Jan 18 22:31:03 xps bluetoothd[2185]: Discover: Connection timed out (110)
Jan 18 22:31:11 xps bluetoothd[2185]: Discover: Connection timed out (110)

After searching for quite some time (a few hours), I ran into this bugreport in Red Hat’s bug tracker, which seemed to somehow match my problem.

I didn’t search for problematic updates as the laptop is brand-new and the headset never worked flawlessly, but looking back (and knowing the solution) it does make sense. After downloading the package from archive.archlinux.org and downgrading bluez from bluez-5.48-1 to bluez-5.47-2 the headset connected immediately. I then added the package to /etc/pacman.conf under IgnorePkg so regular updates are not affected.

I’ll be monitoring the bug-report for changes, hoping for a soonish fix in bluez.

UPDATE: The new release (bluez-5.48-2) fixed the problem according to the bug-report. I installed it on my system, and indeed, the problem is now gone.

Wired headset

After fixing my bluetooth headset (which i use most of the days), I also tried my wired headset (while recharging the bluetooth headset) only to discover that there was a constant background noise (hissing, crackling, …). Again, the problem only manifested itself after Suspend or hibernate, but not when restarting / cold booting.

I could not get to the bottom of this problem yet (I will update this post should i ever find a permanent solution) - but for now I am not 100% sure what the problem is. What i know is:

the noise is only present after resume
the noise is only present in devices connected via 3.5mm Jack
the noise is gone if i reload the kernel-module snd_hda_intel

Changing the power-related settings of the module did not help (with power-saving turned off, the problem still appears after every resume).

The only solution i found so far is to reload the kernel-module (for now manually, whenever i need it) using the following script:

#!/bin/bash
#fix pulseaudio headset crackling

systemctl --user stop pulseaudio
sudo rmmod snd_hda_intel
sudo modprobe snd_hda_intel
systemctl --user start pulseaudio

echo "pulseaudio restarted"

The script stops pulseaudio, removes and reloads the module, and starts pulseaudio again. It produces a fairly loud crack in the headset, and afterwards i need to unplug/replug my headset so it is set as default source, however at least i don’t have to reboot whenever i want my headset to work.

I have not given up on a permanent solution, but this is something that does not bother me too much short-term and i don’t currently want to invest too much time.

New Notebook

Sat, 20 Jan 2018 20:00:00 +0000

I recently decided that after 3 years with my current Notebook, it’s time to get a new one. (No, not one like in the image above) NB: The decision-criteria are based from around Christmas 2017, so newly announced Laptops on the CES have not been considered.

My criteria where as follows:

14’’ or smaller
Ultrabook-style (1.6 kg or lower)
decent (8+) hour battery life
out of the box Linux support for most components
8th generation Intel CPU i5 or i7 (this has proven to be the most complicated
16 G of Ram
Matte Display, non-Touch

Competitors and quick details

For a long time, I had the following candidates on my list:

Lenovo X1 Carbon 2017
Asus Zenbook UX3430UA-GV376T
Dell XPS13 9360R

Lenovo X1 Carbon 2017

Lenovo did disqualify as it did not have the 8th Gen CPU announced yet. They did announce this refresh at CES 2018 - however too late for me. While i did expect this announcement, I had other reasons to go with what i got (you’ll see below why).

Asus Zenbook UX3430UA-GV376T

Asus has the UX3430UA (Matte Display, 14’’ Display in the body of the 13’’ UX330UA, i7-8550U, 16G Ram). While this looks like a great Laptop, most concerns where with Linux support as well as Coil Whine (not on all Devices apparently, so it’s like playing Lottery). The exterior of Zenbooks looks great, however is a fingerprint magnet according to many Reviews of earlier models. As the Case did not really change, this is to be expected from this device. Also, this model was brandnew in December, so not many reviews were available at that time.

Dell XPS13 9360R

I always wanted a XPS 13 since they first announced at CES 2012. However, early models only had very slow CPU’s so they were no option last time around. However, this has changed greatly. With Intel’s release of the Kaby Lake Refresh CPU’s (4 cores in Ultrabook CPU’s), this seemed like the best bet. The only other competitor with an 8thgen CPU and Matte Screen the XPS 13 had in December the Asus Zenbook mentioned above, all other competitors have either a older CPU model (Lenovo X1 Carbon) or a glossy screen (Swift 3, Lenovo Yoga, Huawei X). I did not jump to this Laptop because I already liked the 2012 version (other Manufacturers have nice Laptops too), but thought and researched for about a Month before proceeding with the purchase.

Decision

Knowing that Lenovo would (probably) announce a refreshed version of the X1 Carbon at CES 2018, all 3 had equal chances. Both the Zenbook and the XPS ranged in approximately the same price range (including shipping, with the specs i wanted), however the X1 Carbon (in my desired configuration) would have been ~200$ more expensive. Expecting the refresh in a similar price-range, i was one step closer to taking a decision.

Mid to end-December, the first reviews of Zenbooks with 8th Gen CPU’s appeared. These are important as they showed that battery-life dropped by ~half to the older CPU. Looking at similar reviews about the XPS, these only mention a ~15% drop on battery life compared to the previous model. Charging of the XPS 13 is possible using both the included charger or a USB-C charger (Though only special Charger models which have enough power output). I did not find a clear indication if this is also possible for the Zenbook (which did also influence my decision - as it might mean one fewer charger when traveling). Also, having a 60W Battery compared to a 50W Battery in the Zenbook helped the decision.

I took the Dell XPS 13 9360R (Refreshed with 8th gen Intel CPU) I think my decision was thoroughly researched based on choosing the longer lasting, more linux-proven Notebook (Dell does offer a Developer-edition with Ubuntu for of most XPS models).

Closing

I received the XPS 13 a week ago, first thing i did was setting up Arch-Linux (my distribution of choice). Almost everything worked out of the box with only few problems (some of which will deserve their own Blog-post).

Dell announced a refreshed version of the Dell XPS 13 (the Verge) with some improvements (smaller Case, different colors, different location for the Webcam). However - i don’t expect to be using the Webcam at all - and the new version (Model 9370) only has a 50W Battery, which will reduce the runtime by some.

So far, I did not regret my decision even though a new Model (9370) was announced before i received my Notebook. The Dell XPS 13 was an affordable (even though not cheap) decision and is running smoothly, silently, and (so far) reliably.

Why applying https everywhere is important

Wed, 27 Dec 2017 19:00:00 +0000

Until recently, I did not realize that https is important for ALL sites, not just for sites containing sensitive information, login pages, banking websites, shopping pages pages. The eye-opener for me was Troy Hunts article about this same topic. Troy talks about a Bank having their landing-page http only - while forwarding to https for the login. The problem Troy points out is that the link to the secure login-page can be intercepted / modified / redirected to any other site by anyone able to MitM your connection (ISP, Coffee shop, malicious actor on open Wifi, VPN Provider, …).

While https will not fully solve this (things like ssl strip can still be applied but can be spotted fairly easy), it does increase the complexity for the (malicious) actor to redirect you to the wrong site.

Closing

Honestly, I completely agree with Troy’s post. I did not realize the importance (other than for privacy protection) before his post, but it was eye-opening to me and showed me that https has to be applied everywhere to be effective.

People, apply https everywhere - with Let’s Encrypt it does not cost a penny and is easy to configure for almost every server setup.

duplicity backup - email notifications

Sat, 02 Dec 2017 19:00:00 +0000

This is one post in a series of tutorials on Duplicity. If you are just getting started with duplicity, I recommend to first read the first and second part to get a basic overview of duplicity and get GPG keys setup properly (this will not be covered in this post). You should also visit the duplicity overview post, which contains a list of all posts in this series. The basic configuration of this post will be used in this post to configure duplicity-backup.sh.

Email notifications

When relying on a backup system it is important to be confident that everything worked. The easiest solution to accomplish this is by testing your backup on a regular basis, and by getting notified if something goes astray. The duplicity backup script does offer several notification methods for notifications, email, Slack, pushover, IFTTT and Telegram.

This post will focus on email notification setup as almost everyone uses email, whereas other services have a more specific target audience.

Setup is similar for each of the notification services - simply filling in the variables in the configuration example for the service you’d like to use.

Note: While this covers E-mail notifications, I would recommend to rely on IFTTT for notifications - the setup is simpler than for email as explained in this post.

Email configuration

To have a working email notification, we first need to configure our system to be able to send emails. On Debian, the default is exim4. If you already have your system configured to send emails, then you can skip to the next section, otherwise read on - it won’t take long.

Let’s start by reconfiguring exim4

sudo dpkg-reconfigure exim4-config

The setup will guide us through the configuration. We will only configure outgoing emails and have the server listen on localhost only.

The only 3 settings we need to change are “General type of mail configuration: " to “internet site; mail is sent and received directly using SMTP”.

, “System mail name: " to your domain name and

“IP-addresses to listen on for incoming SMTP connections:” to 127.0.0.1 to make sure the SMTP server is only listening on localhost. Otherwise, we would have an open SMTP server, and it would not take long until it is abused to send Spam emails.

All other settings can remain at default assuming you did not already configure this previously.

Now, we can run a test by sending an email to our email address:

echo "test Email" | mailx -s "test email 1" youremail@gmail.com

Check your mailbox (also the Spam folder) for an email with “test email 1” as subject. You should receive it within minutes if everything is working correctly.

Script configuration

Open the configuration we created in this post and search for EMAIL. Change the following settings to your preference (your email address, and a custom subject if you’d like to change it from the default duplicity-backup Alert ${LOG_FILE}.

EMAIL_TO=youremail@gmail.com
EMAIL_FROM=duplicity@yourdomain.com
EMAIL_SUBJECT=
#EMAIL_FAILURE_ONLY="yes" # send e-mail only if there was an error while creating backup
EMAIL_FAILURE_ONLY="no" # always send emails

I changed the setting EMAIL_FAILURE_ONLY to always send me an email (so i am sure the backup ran). This is also useful to test the settings - if the first email would be sent when the backup fails we cannot be sure sending mails actually works as we never tested it.

Now, run the script again and check your mailbox afterwards.

./duplicity-backup.sh --config duplicity-backup.conf --backup

The script output should end with “Email notification sent to youremail@gmail.com using mailx” If the configuration was correct, you should have received an email. Don’t forget to check the Spam-folder - depending on the email provider the email may be detected as Spam (we did not set the server up to deliver safe, verified emails - so the authenticity of the email is not guaranteed - as the sender address is known (configured above) we can whitelist to get the emails regularly).

No matter what notification you configure - configure at least one of them so you are aware that the backup failed and needs manual interaction / checking.

The best backup-strategy is worthless if it fails to backup silently or the backup is never tested, so please use either this or the other method to get notified of problems with the backup.

duplicity backup - IFTTT notifications

Sat, 02 Dec 2017 19:00:00 +0000

IFTTT notifications

In order to use IFTTT you’ll need an account on the site. If you don’t have one, please register yourself with the service to continue with the tutorial. Go to My applets and click on “New Applet” (or use this link to get to this page directly). Click on the This link, search for “webhooks” and select it. Click on Receive a web request.

Insert an event name (keep note of it - we’ll need this again later) and click on Create Trigger.

Click on That and select a service you’d like to Trigger.

I’m using Send me an email in this case, but feel free to experiment with other notification methods. Click on “Send me an email”. For now, keep all fields identical - you can change the template later if you wish to receive a different subject or content. Confirm the template by clicking on create action, then review and confirm the applet by clicking on Finish in the next screen.

duplicity-backup.sh

Implementing this into the duplicity-backup.sh script requires the configuration from the duplicity-backup.sh post.

Scroll all the way down (or search for “IFTTT”).

# Provider: IFTTT
IFTTT_KEY="" # Key for MAKER channel at IFTTT
IFTTT_MAKER_EVENT="duplicity" # name the event to trigger at IFTTT Maker Channel
IFTTT_HOOK_URL="https://maker.ifttt.com/trigger/$IFTTT_MAKER_EVENT/with/key/$IFTTT_KEY" # ONLY change this if IFTTT changes it
IFTTT_VALUE2="" # general purpose value to pass to your maker channel (optional)

The settings we are interested in are IFTTT_KEY and IFTTT_MAKER_EVENT (and eventually IFTTT_VALUE2 - we can use this to tell us the name of the backup job). IFTTT_MAKER_EVENT will be set to the event name we configured on IFTTT earlier. Next, go to IFTTT Webhook settings to get value for the IFTTT_KEY. You’ll see an URL - the last part of the URL https://maker.ifttt.com/use/c734bdvJ5afSlnwVRukZZ2 (in bolt) is your key and goes to the IFTTT_KEY variable.

Next, search for NOTIFICATION_SERVICE and insert “ifttt” there. The relevant section will then look as follows:

# Possible values for NOTIFICATION_SERVICE are: slack, pushover, ifttt, telegram
NOTIFICATION_SERVICE="ifttt"
NOTIFICATION_FAILURE_ONLY="yes" # send notifications only if there was an error while creating backup

# ...

# Provider: IFTTT
IFTTT_KEY="c734bdvJ5afSlnwVRukZZ2" # Key for MAKER channel at IFTTT
IFTTT_MAKER_EVENT="duplicity" # name the event to trigger at IFTTT Maker Channel
IFTTT_HOOK_URL="https://maker.ifttt.com/trigger/$IFTTT_MAKER_EVENT/with/key/$IFTTT_KEY" # ONLY change this if IFTTT changes it
IFTTT_VALUE2="Server11" # general purpose value to pass to your maker channel (optional)

wrapping up

While from a security perspective, using mail might be seen favorable as it does not involve trusting an external service, the convenience by not having to setup your own mail client on a server (which may not need it for anything else) out weights the concerns. Also, IFTTT is a well established service (in operation since 2011), so i feel safe in trusting them with this little information. No password other than the one to IFTTT has been shared.

Both methods work perfectly fine - however I personally prefer the IFTTT method, as it’s easier to setup as it has no additional requirements. In this post I’ve also shown how to configure email notifications directly, so please choose whichever method seems favorable to you. The most important thing is to be notified should something go wrong with the backup (apart from having a backup!!) - which method is chosen is completely up to the user.

duplicity backup - Google Drive

Sun, 05 Nov 2017 19:00:00 +0000

Google Drive

The duplicity manpage lists multiple ways to use duplicity with Google services:

Google Docs
- Gdata (Legacy backend)
- Pydrive
Google Cloud Storage
pydrive

While Google Docs and pydrive backends save the content to Google Drive (using different methods), which comes for free with any Google account - the Google Cloud Storage backend is their paid service. There is a free tier, however this post will focus on Google drive using pydrive.

Pydrive itself works in 2 ways - by using the regular account (username + password) or with a service account. For now, we will focus on the regular account (it will only save the OAuth-token, not the password / username).

Getting started

The Google Drive backend requires pydrive (the python package) to be installed:

pip install pydrive

We will also need a Google account. Go to the Google developer console and create a new project. Insert a name you’d like (I’m using “duplicity-sync-xmatthias” for this demonstration).

Next, go to the API overview and enable the Google drive API.

After enabling the Drive API, you will be forwarded to the overview of this API.

Click on credentials on the left hand side and configure the OAuth consent screen to your liking. As only you are going to see this screen, it is important to insert something that is meaningful to, but does not need to be complete.

Next, we create a new OAuth Client ID, select other and insert the name for our client (in this case, i used cmd-client.

Now, we are presented with a client-id and a client secret.

(Don’t bother to try the above keys - by the time this post goes live I already deleted the test application)..

We will paste the client-id and client-secret into the pydrive settings file “pydrive.conf” as specified on the duplicity manpage.

client_config_backend: settings
client_config:
    client_id: <Client ID from developers’ console>
    client_secret: <Client secret from developers’ console>
save_credentials: True
save_credentials_backend: file
save_credentials_file: <filename to cache credentials>
get_refresh_token: True

Ours:

client_config_backend: settings
client_config:
    client_id: 76061484583-nfoocmj6e891ml1sbmrq7m4gob4b5qht.apps.googleusercontent.com
    client_secret: 8bXpRP9agTBBZiA7X11X6dKk
save_credentials: True
save_credentials_backend: file
save_credentials_file: /home/matt/pydrivecreds.json
get_refresh_token: True

Now, we need to export the environment variable so duplicity can find this configuration file

export GOOGLE_DRIVE_SETTINGS=/home/matt/pydrive.conf
duplicity full --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB /home/matt pydrive+gdocs://developer.gserviceaccount.com/duplicity_backup_test/

During this first run, we will be asked to visit a Url. As we do so, we are asked to choose an account - and then allow the application to access the Drive folder.

After allowing, we are presented with a code to paste back into the command line.

Press enter and you should see “Authentication successful.”.

Duplicity will now backup and upload the data. After the process is complete, we can check what duplicity did by going to Google Drive and checking the new folder.

To revoke the application access, go to the Google account Permissions page and revoke access to the Application (It should show up under whatever name you gave it in the OAuth screen setup - in my case it is “duplicity-backup”).

duplicity-backup.sh

We can integrate this process fairly easily in the duplicity-backup.sh script this way:

Change DEST to the following:

DEST="pydrive+gdocs://developer.gserviceaccount.com/duplicity_backup_test/"

In addition to this, we also need to export GOOGLE_DRIVE_SETTINGS. We do this by adding the following line right below the DEST line.

export GOOGLE_DRIVE_SETTINGS=/home/matt/pydrive.conf

The remainder of the configuration remains identical to the configuration used in the duplicity-backup.sh post.

We now run the duplicity-backup.sh script.

./duplicity-backup.sh --config duplicity-backup.conf --backup

If everything goes well, Google Drive should show the incremental files as well.

If you still have the cronjob scheduled and used the same configuration file, the backup will automatically go to google Drive instead of the sftp server or a different backend.

Closing

This is yet another method to backup your files to a cloud service securely. Again, the drawback is that the server gains access to your whole Google Drive. Depending on your usage of Google Drive, this may or may not be a problem. As i don’t really use Google Drive for anything else and have plenty of free space available, I am fine with this (The OAuth-tokens saved on the server do only grant access to Google Drive, but to no other Service). Alternatively, you could also create a dedicated account for Duplicity to run the backups.

duplicity backup - Hubic

Tue, 31 Oct 2017 19:20:00 +0000

Hubic

First of all, we need an account at hubic.com (referral link). More than 25 GB will cost you, however for this tutorial the free account will suffice.

Using duplicity with the hubic/Rackspace backend requires the Rackspace CloudFiles Pyrax API library as noted in the duplicity man page. We can install this by running pip install pyrax.

First, we need to login to hubic and go to My account, and then to Developers. We add a new application (Please note that the application-name must be unique - so best use some random characters). Also, we need a redirect url, for which we will use http://localhost/.

We will need the Client ID as well as the Secret Client key from the Application we just created. and insert that in the file ~/.hubic_credentials. The file should look as below - substitute all variables right of the = sign with the values in your Developers console.

[hubic] 
email = emailididregisterwith@gmail.com 
password = VerySecureRandomOnlinePassword11 
client_id = api_hubic_mVZdgTqLb3ty3BAJZn5Nyk8zBGrN4KVX 
client_secret = QJOvuZe0gHg13ilM3DpCCaNdIlnRUTDa01pbAH5jCfx5BpkpZiu4IAnsuuAnn3Mr 
redirect_uri = http://localhost/

(Don’t bother to try the above keys - by the time this post goes live I already deleted the test application).

As the API requires us to add the password in clear text, we make sure the file is only readable to the user himself by issuing a chmod 600 ~/.hubic_credentials.

Running a first manual test: duplicity --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB --ssh-askpass /tmp/backupTest/ "cf+hubic://duplicitybackup1"

This will result in the folder /tmp/backupTest/ to be synchronized to hubic - in the backup-folder duplicitybackup1. In the web-console of hubic, we go to “my backups” - where we can see the below backup-files.

As mentioned above, up to 25 GB is free - after that additional storage is available for paying customers.

duplicity-backup.sh

Implementing this into the duplicity-backup.sh script is as easy as setting the “DEST=” variable to “cf+hubic://duplicityBackup1”.

DEST="cf+hubic://duplicityBackup1"

The rest of the configuration remains identical to the configuration used in the duplicity-backup.sh post.

We now run the duplicity-backup.sh script.

./duplicity-backup.sh --config duplicity-backup.conf --full

If you still have the cronjob scheduled and used the same configuration file, the backup will automatically go to hubic instead of the sftp server or a different backend.

Drawbacks

Hubic - the service

Hubic is pretty slow - uploading the simple test-folder took quite some time (they seem to implement fairly aggressive rate-limiting. Also, bigger files may not be allowed in the free version. This is not clearly visible - but the next level lists “No maximum file size.” as an advantage of the Paid version, suggesting that the free version has some limit applied.

Security

As the password to hubic needs to be saved in clear text in a configuration file (on a server, eventually), it is important to use a unique password, so an attacker would eventually gain access to only this account, and none other. Also, if duplicity is running on a server (which is a common usecase), you should consider using a dedicated account for backups as well as backing up to another Service as well. Should the server be breached, one needs to assume the hubic-account also compromised - so an attacker would be able to log in to hubic and delete the backups - essentially making recovery impossible.

Therefore, I cannot fully recommend using hubic as a backup solution for Servers. Backing up Pictures from your own PC (in addition to other backups!!!) will probably be fine as the data is encrypted with your GPG key.

FULL DISCLOSURE If you register with hubic using the link above, I will get free space added through the referral program. If you want to avoid this, please use this link or type in https://hubic.com into your browser manually.

duplicity backup - Microsoft Onedrive

Mon, 30 Oct 2017 19:00:00 +0000

Microsoft Onedrive

In order to get going with Microsoft Onedrive, we need a Microsoft Account. The free version offers 5Gb of free storage. Older accounts may have more storage available - more is available for paying customers.

Using the Onedrive backend requires python-requests and python-requests-oauthlib to be installed (as defined on the man page under Requirements). Let’s simply install them on our system.

sudo apt-get install python-requests python-requests-oauthlib

If you skip this step and don’t have these packages installed, then the execution will fail with the following message: BackendException: OneDrive backend requires python-requests and python-requests-oauthlib to be installed. Please install them and try again.

Let’s run an initial (manual) test:

duplicity --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB  /tmp/backupTest/ "onedrive://duplicityTest1/backup"

As this is the first time, we will need to go to the link the program shows to confirm the OAuth request.

After confirming by clicking on yes, we need to copy/paste the URL we are forwarded to into duplicity which is waiting/asking for it. Duplicity will now save the oauth-key, so no further actions are required. Should you ever want to revoke access to your onedrive account, simply go to the apps and services page and remove access to duplicity.

duplicity-backup.sh

Implementing this into the duplicity-backup.sh script is as easy as setting the “DEST=” Variable to “onedrive://duplicityTest1/backup”. The rest of the configuration remains identical to the configuration used in the duplicity-backup.sh post.

DEST="onedrive://duplicityTest1/backup1"

Please note that it is necessary to run the first run manually to get OAUTH setup, otherwise the backup-script will not work as duplicity can only log you in if it is run interactively.

We now run the duplicity-backup.sh script again.

./duplicity-backup.sh --config duplicity-backup.conf --full

If you still have the cronjob scheduled and used the same configuration file, the backup will automatically go to onedrive instead of the sftp server.

Drawbacks

access level to your onedrive Folder

One downside of Onedrive is that currently you need to grant duplicity access to all your files. I don’t expect duplicity to exploit this, however, having the possibility to grant a single, dedicated folder like it is possible with Dropbox would be desirable. Especially if you intend to run duplicity on a server - the OAuth key will need to be saved on that server - which could be a potential problem if sensitive files are in Onedrive and your server is be breached. Fixing this should be easy - the new API does support this, but it does not seem to be implemented into the duplicity backend (yet), which is still using the old skydrive API.

Office 365

I also have to say that I did only test onedrive with a personal account, access to a Office 365 Onedrive account may not work (Onedrive personal and Office 365 are called identically, but are 2 separate things when interacting with them on an API level).

FULL DISCLOSURE If you register with onedrive using the link above, i will get free space added through the referral program. If you want to avoid this, please use this link or type in https://onedrive.com into your browser manually.

duplicity backup - duplicity-backup.sh

Thu, 26 Oct 2017 20:00:00 +0000

Intro

In part 1, we’ve seen how to generate GPG keys and getting started with duplicity. Part 2 covered moving data off the server using sftp and scheduling the jobs using cron.

In this part, we’re going to use an open source duplicity script which will simplify scripting backups as well as eventually restoring files. Until now, we needed to remember the full command line of the duplicity command. This makes handling fairly difficult. The script has a very good readme page - so i recommend to dig deeper into the different options on your own.

New automation script

In order to get the script, we will be using git. If git --version shows “command not found”, install git with sudo apt-get install git. Next, we need to clone the github repository and copy the configuration-file example to our own configuration.

git clone https://github.com/zertrin/duplicity-backup.sh
cd duplicity-backup.sh
git checkout master
cp duplicity-backup.conf.example duplicity-backup.conf

The script gives us a simple configuration file where all options can be set. The logic is in another file.

Open the configuration file with your favorite editor and modify the variables ROOT, DEST, PASSPHRASE, INCLIST, GPG_ENC_KEY, GPG_SIGN_KEY, STATIC_OPTIONS, LOGDIR.

The relevant content of the files should look as follows:

# Folder to backup
ROOT="/tmp/backupTest/"
# Destination
DEST="sftp://matt@backupserver:1234//home/matt/offsitebackup"

# Password for ssh key
FTP_PASSWORD="SSHTest1!"

INCLIST=( "*" )
EXCLIST=( "" )

#Passphrase for Signing key
PASSPHRASE="TestSig1!"
# Encryption key
GPG_ENC_KEY="3E988E6866B39EE1"
# Signing key
GPG_SIGN_KEY="E24E7891636093DB"

STATIC_OPTIONS="--full-if-older-than 14D --s3-use-new-style"

LOGDIR="/var/log/"
LOG_FILE="duplicity-$(date +%Y-%m-%d_%H-%M).txt"
LOG_FILE_OWNER="root:root"

There are a lot of other options in the configuration - most have sensible defaults (keeping a total of 4 full backups and cleanup older ones, logging, …). Please note that this tutorial is based on the master branch as it was on github on Oct 23rd 2017. The default configuration may change over time as this is a still actively developed project.

After making the above changes, let’s try to run the script. The script has pretty good error-checking - making sure folders exist and the permissions are correct.

./duplicity-backup.sh --config duplicity-backup.conf --full

Depending on how often the backup has been tested before, we may have different outcomes on the cleanup-section. In my case, there were multiple full backups to delete as i tested full backups very often initially.

Now, we’ll replace our crontab-entry we made last time with the new entry:

#old:
#0 1 * * * root /root/backupscript.sh >> /var/log/backupscript.log 2>&1
#new:
0 1 * * * root /root/duplicity-backup.sh/duplicity-backup.sh --config /root/duplicity-backup.sh/duplicity-backup.conf >> /var/log/backupscript.log 2>&1

The script simplifies our life a lot as otherwise we would need to always specify the exact path to the backup location (which can be very frustrating if the remote path is a Amazon S3 bucket or something similar). It would probably be OK for scheduling - but as soon as you need to recover a file, running duplicity-backup.sh [-c config_file] --restore-file backup2/file1 will be a life-changer.

Backup script configuration

Run ./duplicity-backup.sh --config /root/duplicity-backup.sh/duplicity-backup.conf --backup-script to backup the configuration and GPG keys - so the restore-process can be migrated to another server easily. This process will ask you for both key passwords, as well as a backup-password (Yes, the backup will be encrypted as well) and show the decryption-instruction on screen.

Wrapping up

When configuring the backup, always test the backup at least once (best regularly) by restoring to another folder. It’s not sufficient to have a backup-process - it’s equally important to know that the restores will work as well. Also, consider setting up one of the mail options - the script can send mails when the process failed - or each time it ran - so you are notified should something go wrong. Otherwise, as long as nothing happens, backups may run in unnoticed in the background for months - only to discover afterwards that there was a problem and the backup did not run in the last days/weeks/months.

I may cover setting up email notifications in a future blogpost.

In the next part, we will take a look at some of the different cloud-based-options duplicity offers and how to configure them with the duplicity-backup script.

duplicity backup - remote backup

Sun, 22 Oct 2017 21:15:00 +0000

Intro

In the first part of the duplicity series, we saw how to create GPG keys, create encrypted and signed backups locally and restore them. Now, we are going to move data off the server (so we have an offsite-backup). Also, we will schedule the backup using cron so the backup is run nightly.

I like to manually run the commands before scripting stuff, so i fully understand what is happening and can discover problems before the script is failing daily. Therefore, the script implementation will be the last step.

Note: As in the first part, this tutorial will use root on the local system. On the remote system, the user “matt” will be used.

sftp

First, let’s make sure we can connect to the backup-server:

# ssh backupuser@backupserver [-p port]

If you have not yet created the SSH keys (we will do this in the next step) - then this should ask for the password of the remote user.

Next - let’s try to run a full backup.

# duplicity --encrypt-key [encrypt-key] --sign-key [sign-key] --ssh-askpass [folder-to-backup] sftp://[user]@[backupserver]:[port]//[folderonremotehost]

duplicity --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB --ssh-askpass /tmp/backupTest/ sftp://matt@backupserver:1234//home/matt/offsitebackup

Please note that the port can be omitted if the default SSH port (22) is used. This will again ask for the remote users password, as well as the password for the Sign-key, so we need to be carefull when we insert which password.

Restore / verify operations follow the same logic as explained in the first part but by exchanging the target plugin file://[targetfolder] with sftp://[targetfolder]. To check that the backup was indeed made, SSH into the backup server and check the backup-folder (ls -l /home/matt/offsitebackup)

Now this was easy - now let’s wrap this into a little script which can run automatically.

SSH Keys

For the following scripts to work, we will need password-less SSH setup. This is as easy as running the following command and accepting all defaults. As this key allows everyone in possession of the key to login to the remote server, we will use the password SSHTest1! as password for the key. As always, this is a sample password with the intend to be easy and clear to the reader, in reality I am using a much stronger password.

ssh-keygen -b 4096 -t rsa

After the key is generated, we copy the key to the target server with the following command.

#ssh-copy-id [user]@[targethost] -p [port]
ssh-copy-id matt@backupserver -p 1234

Now, we test access using the key:

[user]@[targethost] -p [port]
ssh matt@backupserver -p 1234

If everything is working correctly, you should see the welcome message from the backup server.

First script (selfmade)

We will now put what we learned into a very simple script. In the next part, we will switch to a better version of the script - but for now we do it “all by ourselves”. Open your favorite editor and paste the following script to a new file called backupscript.sh. Please use your secure password and key-id’s you generated in part 1.

#/bin/bash

# passwod to ssh key
export FTP_PASSWORD="SSHTest1!"
# password to the encrypt key - set to empty so duplicity is not asking
export PASSPHRASE=""
# password to the Sign key
export SIGN_PASSPHRASE="TestSig1!"

ENCRYPT_KEY=3E988E6866B39EE1
SIGN_KEY=E24E7891636093DB
# Local source folder
SOURCE="/tmp/backupTest/"
# (remote) target folder
TARGET="sftp://matt@backupserver:1234//home/matt/offsitebackup"

duplicity --full-if-older-than 5D --encrypt-key ${ENCRYPT_KEY} --sign-key ${SIGN_KEY} ${SOURCE} ${TARGET}

Make the file executable by running chmod +x backupscript.sh. To keep the passwords to ourselves (the root user) - we also modify the permissions to disallow everything from other users chmod 700 backupscript.sh.

In the script, I also added the option --full-if-older-than 5D - which we did not use before and has duplicity do a full backup if the last full backup is older than 5 days.

let’s run the script

./backupscript.sh

output:

./backupscript.sh 
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sat Oct 21 10:54:06 2017
--------------[ Backup Statistics ]--------------
StartTime 1508576169.90 (Sat Oct 21 10:56:09 2017)
EndTime 1508576169.96 (Sat Oct 21 10:56:09 2017)
ElapsedTime 0.06 (0.06 seconds)
SourceFiles 61
SourceFileSize 45086 (44.0 KB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 0
RawDeltaSize 0 (0 bytes)
TotalDestinationSizeChange 732 (732 bytes)
Errors 0
-------------------------------------------------

As we can see, the backup completed successfully. We can try a restore (documentation on that is in part 1).

Schedule

It is important to run backups regularly - therefore we will now use “crontab” to schedule the backup process. There are 2 possibilities to schedule tasks with crontab - either by user crontab -e or in the file /etc/crontab. I will be using /etc/crontab for this tutorial. This means, only super-users (root) can change the schedule (or disable backups) - however we can still define which user the backup should run as. As the initial explanation in the file says:

Unlike any other crontab you don’t have to run the `crontab’ command to install the new version when you edit this file and files in /etc/cron.d. These files also have username fields, that none of the other crontabs do.

Let’s open the crontab file (we need to be root for this, so prefix the command with sudo). and add the following line at the end:

# m h dom mon dow user	command
# 0 1 * * * [user] [full_path_to_backup_script] >> [path_to_log_file] 2>&1

0 1 * * * root /root/backupscript.sh >> /var/log/backupscript.log 2>&1

By using 0 1 * * * we run this backup-script daily at 1 am. Also, we specify /var/log/backupscript.log as log-file. We can now wait overnight and then check the log-file. The content will be similar to the output above.

Conclusion

This was fairly simple - however haven’t reached our final goal yet. Remember, we wanted to have 3 copies of our data, in at least 2 different locations, with at least 1 backup offsite. Until now, we either have a backup on the same server (which will not help much in case of a server crash) - or a backup at a remote server (which may be inconvenient if we delete a wrong file by accident).

In the next part, we will switch from our home-made quick and dirty script to a open source duplicity-backup script which gives us more options out of the box without the need to reinvent the wheel. Also, we will have a look at the “multi” duplicity plugin, which allows us to specify multiple targets for the same backup.

duplicity backup - getting started

Tue, 17 Oct 2017 20:01:00 +0000

Intro

A basic rule for Backups is the 3-2-1 rule: 3 total copies of the data - on 2 different mediums, at least 1 offsite.

In the following multi-part tutorial I will show how to generate fairly secure backups with duplicity following the above strategy.

Note: My test-system is debian-based - however the commands should also work with most other Linux systems by replacing the name of the package-manager to the system-default. The tutorial will assume that you have root access on the system. Most steps are also possible as regular user - however full system backups are not possible in this case.

About duplicity

Duplicity is a python-based backup-program which has multiple backends. It uses encryption by default and handles the gpg encryption for us. The signature-key defaults to the encryption-key - in that case, the encryption-key is used for signing as well. I strongly recommend to create a separate Keypair - the following will assume you will do the same.

Encryption and keys

Duplicity is able use 2 different methods of encryption.

symmetric encryption
asymmetric encryption

This post will focus on asymmetric encryption as this is considered more secure than symmetric encryption (it uses a different key for encryption and decryption). Explaining the cryptographic details would span a whole book, so I’ll leave further research to the interested reader as there are many good resources on this topic available.

To ideally protect our backup, we will be generating 2 GPG key-pairs:

encryption key pair (confidentiality)
signing key pair (integrity)

While backing up, the public key of the encryption key pair is used to encrypt the data, while the private key of the signing key pair will be used to ensure the integrity of the backup.

We will use separate keys as the pass-phrase for the signing key needs to be available when running the backup (which is normally a script or scheduled job).

Let’s first make sure we have enough entropy for the key generation.

generating keys - entropy

Note: the key-generation may take a while, depending on the available entropy in the system (check using cat /proc/sys/kernel/random/entropy_avail). On a fresh test-system I faced this problem, while on my regular PC i had enough entropy available.

The following should be run in a separate session and generate enough entropy

apt-get install rng-tools
rngd -f -r /dev/urandom

Generating the keys

For this tutorial, I will use the passwords TestEnc1! for the Encryption key (named duplicity_enc), and TestSig1! for the signing key (named duplicity_sign). In reality, a random, secure password/pass phrase should be used (this passwords have multiple problems - please never use such passwords for anything). Also, please consider using 4096 bit keysize - this will increase security another bit (i did stick with defaults for this tutorial - but my production implementation does use 4096 bit keysizes).

We will need to run the following command twice, once for the encryption key pair and once for the signing key pair. Make sure to use different passwords, otherwise a single key could be used as well.

Note that the signing-key needs to be generated first.

gpg --full-generate-key

sample output:

gpg (GnuPG) 2.1.18; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 
Key does not expire at all
Is this correct? (y/N) y

GnuPG needs to construct a user ID to identify your key.

Real name: duplicity_sign
Email address: 
Comment: Test key for duplicity signing
You selected this USER-ID:
    "duplicity_sign (Test key for duplicity signing)"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: key E24E7891636093DB marked as ultimately trusted
gpg: revocation certificate stored as '/root/.gnupg/openpgp-revocs.d/5BA6FC44F5C7FBF5CCB62A43E24E7891636093DB.rev'
public and secret key created and signed.

pub   rsa2048 2017-10-15 [SC]
      5BA6FC44F5C7FBF5CCB62A43E24E7891636093DB
uid                      duplicity_sign (Test key for duplicity signing)
sub   rsa2048 2017-10-15 [E]

After generating both keys, the output of gpg --list-keys --keyid-format long should look similar to mine:

/root/.gnupg/pubring.kbx
-----------------------------
pub   rsa2048/E24E7891636093DB 2017-10-15 [SC]
      5BA6FC44F5C7FBF5CCB62A43E24E7891636093DB
uid                 [ultimate] duplicity_sign (Test key for duplicity signing)
sub   rsa2048/1118A69B4088EC29 2017-10-15 [E]

pub   rsa2048/3E988E6866B39EE1 2017-10-15 [SC]
      1FD28DA009BBEE82A543B2CC3E988E6866B39EE1
uid                 [ultimate] duplicity_enc (test key for duplicity encryption)
sub   rsa2048/74DCEA71B0217B9B 2017-10-15 [E]

The interesting part is E24E7891636093DB and 1118A69B4088EC29, which contains our key-id’s. The key-id’s will be unique for everyone running though this tutorial - the keys must be replaced with your own for every step they appear in.

Next, we will need to sign the encryption key.

gpg --sign-key duplicity_enc

output:

sec  rsa2048/3E988E6866B39EE1
     created: 2017-10-15  expires: never       usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa2048/74DCEA71B0217B9B
     created: 2017-10-15  expires: never       usage: E   
[ultimate] (1). duplicity_enc (test key for duplicity encryption)


sec  rsa2048/3E988E6866B39EE1
     created: 2017-10-15  expires: never       usage: SC  
     trust: ultimate      validity: ultimate
 Primary key fingerprint: 1FD2 8DA0 09BB EE82 A543  B2CC 3E98 8E68 66B3 9EE1

     duplicity_enc (test key for duplicity encryption)

Are you sure that you want to sign this key with your
key "duplicity_sign (Test key for duplicity signing)" (E24E7891636093DB)

Really sign? (y/N) y

backup GPG keys

Export public keys and store them in a safe location (if you loose them you cannot get your files back).

# gpg --armor --export -a [keyid] > [public_key_filename].key
gpg --armor --export -a 5BA6FC44F5C7FBF5CCB62A43E24E7891636093DB > duplicity_sign_public.key
gpg --armor --export -a 1FD28DA009BBEE82A543B2CC3E988E6866B39EE1 > duplicity_enc_public.key

Export public keys and store them in a safe location. This will ask you for the passwords/pass phrases provided above.

# gpg --armor --export-secret-keys -a [keyid] > [private_key_filename].key
gpg --armor --export-secret-keys -a 5BA6FC44F5C7FBF5CCB62A43E24E7891636093DB > duplicity_sign_private.key
gpg --armor --export-secret-keys -a 1FD28DA009BBEE82A543B2CC3E988E6866B39EE1 > duplicity_enc_private.key

Details about the gpg key restore process may be shared in a later post.

install duplicity

apt-get install duplicity

Now let’s create a few test folders with 5 files in each folder.

mkdir backupTest
cd backupTest
mkdir backup{1..10}
touch backup{1..10}/file{1..5}

backupTest
|-- backup1
|-- backup1/file1
|-- backup1/file2
|-- backup1/file3
|-- backup2/file1
...

first test

To start, we’ll backup the folder /tmp/backupTest to /tmp/backup. This will ask for the password to the signing key (we are ready to script the backup - let’s test things manually for now).

# duplicity full --encrypt-key [encryptkey] --sign-key [signkey] [folder] file://[backupfolder]
duplicity full --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB /tmp/backupTest file:///tmp/backup

Note that the target-folder has 3 “/” - this is because I’m using absolute paths here.

To verify the backup run duplicity verify (Target folder and folder are reversed). This will ask for the password of our encryption key.

# duplicity verify --compare-data --encrypt-key [encryptkey] --sign-key [signkey] file://[backupfolder] [folder] 
duplicity verify --compare-data --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB file:///tmp/backup /tmp/backupTest

--compare-data compares the actual content of the files, not just the files themselves. For remote connections and big backups this is not recommended.

Now, lets change a file echo test > backup2/file2 and rerun the above verify command.

Now the output should show a difference.

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Oct 15 20:14:03 2017
GnuPG passphrase for decryption: 
Difference found: File backup2/file2 has mtime Sun Oct 15 20:15:15 2017, expected Sun Oct 15 13:10:53 2017
Verify complete: 61 files compared, 1 difference found.

The 1 difference found highlights the difference we just created. Running the incremental sync again will remove this difference again.

# duplicity incr --encrypt-key [encryptkey] --sign-key [signkey] [folder] file://[backupfolder]
duplicity incr --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB /tmp/backupTest file:///tmp/backup

output:

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Oct 15 20:14:03 2017
GnuPG passphrase for decryption: 
GnuPG passphrase for signing key: 
--------------[ Backup Statistics ]--------------
StartTime 1508091392.03 (Sun Oct 15 20:16:32 2017)
EndTime 1508091392.11 (Sun Oct 15 20:16:32 2017)
ElapsedTime 0.07 (0.07 seconds)
SourceFiles 61
SourceFileSize 45086 (44.0 KB)
NewFiles 0
NewFileSize 0 (0 bytes)duplicity-backup.conf_mine
DeletedFiles 0
ChangedFiles 1
ChangedFileSize 10 (10 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1
RawDeltaSize 15 (15 bytes)
TotalDestinationSizeChange 833 (833 bytes)
Errors 0
-------------------------------------------------

Now we have one full backup - and one additional partial backup.

looking into our backup-folder we see the following:

-rw------- 1 root root 1758 Oct 15 20:14 duplicity-full-signatures.20171015T181403Z.sigtar.gpg
-rw------- 1 root root 1001 Oct 15 20:14 duplicity-full.20171015T181403Z.manifest.gpg
-rw------- 1 root root 1527 Oct 15 20:14 duplicity-full.20171015T181403Z.vol1.difftar.gpg
-rw------- 1 root root  823 Oct 15 20:16 duplicity-inc.20171015T181403Z.to.20171015T181625Z.manifest.gpg
-rw------- 1 root root  833 Oct 15 20:16 duplicity-inc.20171015T181403Z.to.20171015T181625Z.vol1.difftar.gpg
-rw------- 1 root root  846 Oct 15 20:16 duplicity-new-signatures.20171015T181403Z.to.20171015T181625Z.sigtar.gpg

We see that we have one full sync - and one incremental sync.

restore

Testing the backup (aka restoring) is important to be sure the backups are working. Restoring with duplicity is as simple as inverting the backup-command (without full / incr keyword) and supplying the password to the encryption key. For completeness, we will include the restore-keyword to make the intent obvious.

My restore will go to the path /tmp/test2. Restoring to a non-empty path will result in the message “Will not overwrite.”.

# duplicity restore --encrypt-key [encryptkey] --sign-key [signkey] file://[backupfolder] [folder] 
duplicity restore --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB  file:///tmp/backup /tmp/test2

This folder now contains the same content than /tmp/backupTest. The restore-process also restores file permissions - which is very handy for full system backups.

Final note

While testing the encryption I ran into some unexpected problems:

i used the wrong key-id (wrong format) for encryption, so decryption was not possible anymore. Instructions i found on the Internet were not very clear in this regard.
i thought that the restore / decryption would not require a password (which would be a big no go). The problem turned out to be caching of the GPG agent, which had the file loaded after the first run.

In total, the above problems cost me about an hour. In the next part, we will look at offsite-backups (transferring the backup to another server).

Hope you learned some

Matthias

Security considerations while blogging

Tue, 10 Oct 2017 20:10:00 +0000

Blogging about the blog/server-setup is a fairly difficult topic for me.

On one side, I’d like to share how i set things up so others can get started easily (and hopefully rather securely), while on the other side, I fear that going into too much depth will open up my site / infrastructure to attacks as this simplifies information gathering. I would not want to deliver my site to attackers on a silver plate - at least have them work for the informations. In the end, most bad guys are financially motivated - once the cost of attack is higher than the benefit, most will look for easier targets.

how I will handle it in this blog

I will try to keep the balance (sharing as much as possible, without exposing all details). Decisions about which details i will write about, and which i will only touch slightly will be made case by case (or post by post-basis).

I’ll also keep it as transparent as possible if all details are posted or not.

I hope my posts will still help some people get started securely.

cheers Matthias

Switching keyboard layout - quick update after 5 days

Tue, 03 Oct 2017 22:00:00 +0000

As I pointed out in my post last week, I switched keyboard layout to english on all my machines.

So far it’s been going better than expected - I only thought about reverting back once while beeing in a rush and constantly missing the same symbol (semicolon) - however resisted the urge and am now mostly confident in what I type. It still requires some concentration to not fall back to the layout I’m used to - however it’s far less obstructive than I immagined.

That’s it for today, read you soon

Switching keyboard layout

Thu, 28 Sep 2017 20:20:12 +0000

A colleague of mine recently told me

“You as a developer should consider switching to english keyboard layout - most programming languages have probably been developed by people with english keyboard-layouts - so language-specific buttons will be reachable much easier than on the german keyboard.”

Having used a german keyboard most of my life I could not deny a certain logic to it. Also - It’s been a while since I got used to switching layouts from time to time (default keyboard on a newly installed linux is english - and it helps to be able to get out of vi without googling “where is : on the UK keyboard”).

As I’m almost never writing in german anyway it should not matter much to loose the “Umlaute” - but help me in the long run to type faster. With this background I decided this morning to switch to the UK keyboard layout (the layout is identical to german - but most programming-specific keys are reachable without 3-button combos). So far it’s working mostly fine - just y/z are sometimes swapped - I guess i can get used to this layout within another day or 2. I still keep the layout-table open in a browser tab (i only swapped the layout - not the physical keyboard) so i can have a quick look if i can’t find a key.

I’ll try and post an update later on if I could successfully stick with it or if i reverted back to my comfort zone.

Check if shell-script is runing with root permissions (or with sudo)

Sun, 24 Sep 2017 11:31:48 +0000

Some scripts need to run as root (iptables for temporary rules, installer scripts, …). A quick way to check if the shell-script is running as root or with sudo (sudo should be the preferred way) is as follows.

if [ ! ${EUID} -eq 0 ]; then
  echo "Please run as root or with sudo"
  exit
fi

this has been tested tested on debian-based and arch-linux, but should also work on most other linux derivates.

it’s not new - but i was not aware of it until recently.

Compiling python 3.6 for centos 5.11 with openssl

Mon, 18 Sep 2017 18:31:00 +0000

Lately I was about to develop and deploy a python-script to a company-server (internal and firewalled) which is still running Centos 5.11.

My first reaction was “can we upgrade to a newer system” - however at this time this is not an option.

Before doing this on the server, i ran multiple tests in a docker-image as this server does not have a test-instance.

docker run -it centos:centos5 /bin/bash

As centos5 has reached End of Life, some tweaking was needed to get yum in the container to behave as expected as the repositories were moved to “vault”.

There are cleaner/better ways to do this - but it works for this demonstration.

mkdir /var/cache/yum/base/ \
&& mkdir /var/cache/yum/extras/ \
&& mkdir /var/cache/yum/updates/ \
&& mkdir /var/cache/yum/libselinux/ \
&& echo "http://vault.centos.org/5.11/os/x86_64/" > /var/cache/yum/base/mirrorlist.txt \
&& echo "http://vault.centos.org/5.11/extras/x86_64/" > /var/cache/yum/extras/mirrorlist.txt \
&& echo "http://vault.centos.org/5.11/updates/x86_64/" > /var/cache/yum/updates/mirrorlist.txt \
&& echo "http://vault.centos.org/5.11/centosplus/x86_64/" > /var/cache/yum/libselinux/mirrorlist.txt

Installing dependencies:

yum install -y gcc gcc44 zlib-devel python-setuptools readline-devel wget make perl

After some trial and error i discovered that openssl is necessary to even get pip to work properly (it requires openssl unless you want to use “–trusted-host”).

To get python to compile correctly with openssl, compiling openssl with a specific flag was necessary (please note that openssl 1.0.xx was used as openssl 1.1.xx causes perl-dependencies on this legacy system).

openssl

cd /tmp
# download openssl - please check https://www.openssl.org/source/ for the latest 1.0.21 version.
wget https://www.openssl.org/source/openssl-1.0.2l.tar.gz
tar xzvpf openssl-1.0.2l.tar.gz
cd openssl-1.0.2l
./config --prefix=/usr/local/ssl --openssldir=/usr/local/ssl
#modify Makefile to include -fPIC in CFLAGS # similar to export CFLAGS=-fPIC
sed -i.orig '/^CFLAG/s/$/ -fPIC/' Makefile
make && make test && make install

python

##python:
cd /tmp
MAJORVERS=3.6
VERSION=${MAJORVERS}.1

# https://superuser.com/questions/646455/how-to-fix-decimal-module-compilation-error-when-installing-python-3-3-2-in-cen
export CC=/usr/bin/gcc44


wget https://www.python.org/ftp/python/${VERSION}/Python-${VERSION}.tgz
tar xzvf Python-${VERSION}.tgz
cd Python-${VERSION}

To enable openssl in Modules/Setup.dist (based on this instruction)

Change the following section from this

#_socket socketmodule.c

# Socket module helper for SSL support; you must comment out the other
# socket line above, and possibly edit the SSL variable:
#SSL=/usr/local/ssl
#_ssl _ssl.c \
# -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
# -L$(SSL)/lib -lssl -lcrypto

# Socket module helper for socket(2)
_socket socketmodule.c

# Socket module helper for SSL support; you must comment out the other
# socket line above, and possibly edit the SSL variable:
#SSL=/usr/local/ssl
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
-L$(SSL)/lib -lssl -lcrypto

build and install as alternative version
./configure --prefix=/opt/python${MAJORVERS}
 make altinstall

# link the alternative binaries into a location in $PATH
ln -s /opt/python${MAJORVERS}/bin/python${MAJORVERS} /usr/local/bin/python${MAJORVERS}
ln -s /opt/python${MAJORVERS}/bin/pip${MAJORVERS} /usr/local/bin/pip${MAJORVERS}

run your newly installed python:

/opt/python3.6/bin/python3.6

Try to import the ssl module - if this works then python with openssl is installed.

import ssl

This guide is far from perfect - but so is having to deal with Centos5 in 2017. It installs python3.6 with openssl in Centos5.

Hope this helps someone passing by here.

Using App.xaml in .dll-assembly

Wed, 13 Sep 2017 10:30:00 +0000

I’m currently working on a wpf-application cooperating with another company. We have only part of the sourcecode and therefore only compile a few .dll’s, the main executable however get’s shipped to us as finished product.

The application uses style-files that get loaded at startup by the main application from files in a specific folder, therefore we faced a the problem that these styles don’t get applied at design-time, but only at runtime.

The Problem:

While changing tons of stuff for a new UI we are working on I started using a new user-control (which derrives from button) in order to have a consistant layout all over the app. The default style sadly had the height set to “auto”, which caused the layouts to be very weired during design-time, but work at runtime (because of the applied style which sets the height to the desired value). Fixing this was a priority as our design-team struggled with the messed up designs.

To fix this i had tree possibilities - eighter set the correct height for each button manually (which the design-team started to do), include a style at each control we’re editing, or load resources in an App.xaml. Hardcoding the height is neighter elegant nor usable as i suspected the size of the buttons to be changed at a later point (and i was proven right later on).

I decided to go for the last approach as this seemed to be the easyest and fastest. However - we’re not compiling the main executable, only a .dll (which does not have a app.xaml where you can define the styles).

The Solution

Seaching the web i found a working solution at The team blog of the Expression Blend and Design products. The team blog of the Expression Blend and Design products. That page suggests that using App.xaml is possible for .dll-files at designtime (using them only during design-time, and excluding them at compile-time). To get this working well for us i had to make a few changes as we don’t use one style-file but have the styles and themes separated in many files and load them as resources, using it as shown in the example would have caused lots of issues when adding new files. however the principle is quite easy and easily understandable:

open the .csproj of your project and insert the following:

  <ItemGroup Condition="'$(DesignTime)'=='true' AND
              '$(BuildingInsideVisualStudio)'!='true'
               AND '$(BuildingInsideExpressionBlend)'!='true'">
    <Resource Include="Skins\DesignTimeTheme.xaml">
      <Generator>MSBuild:Compile</Generator>
      <SubType>Designer</SubType>
    </Resource>
    <Compile Include="App.xaml.cs">
      <DependentUpon>App.xaml</DependentUpon>
      <DesignTime>true</DesignTime>
    </Compile>
  </ItemGroup>

Explanation:

Adding the condition to the ItemGroup applies that to all resources that lie below. That means, i have the App.xaml and the Theme there only at design-time, but they’ll be excluded at runtime. The advantage of this is that upon release, the resources are not there several times (once for the application and once for the library).

Now, every new style-file is added into this ItemGroup.

Summary:

Using the above, i was able to load my Styles in app.xaml (as you’d usually do with a wpf-project) - but doing this for the assembly (without creating a fake executable or so).

The solution may not be perfect, however it prooved to be quite useful to our designers which had to modify some styles.

hello world

Sun, 03 Sep 2017 11:30:00 +0000

I’ve been thinking long about starting a blog and finally decided to just try and start one.

I can’t promise I’ll be posting regularily - however I’ll try to post whenever i have something new or exciting to write about.

Most of my interrests are around tech, so the content will be mostly tech.

We’ll see where this brings me.

so long, Matthias