Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Album filter support - needs work #63

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Album filter support - needs work #63

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
76a9d2c
replace ? to _ in filenames
Nov 25, 2024
b961e9e
Added album_filter support
Nov 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ SMGMG_BK_USER_TOKEN = "<User Token>"
SMGMG_BK_USER_SECRET = "<User Secret>"
SMGMG_BK_DESTINATION = "<Backup destination folder>"
SMGMG_BK_FILE_NAMES = "<Filename with template replacements>"
SMGMG_BK_ALBUM_FILTER = "<Only download albums matching the filter>"
```

| Configuration identifier | Required | Default Value | Description |
Expand All @@ -75,6 +76,7 @@ SMGMG_BK_FILE_NAMES = "<Filename with template replacements>"
| store.force_video_download | No | false | When true, videos are downloaded also if marked as "under processing" |
| store.concurrent_albums | No | 1 | Number of concurrently analyzed albums. It multiplies API calls, don't stress this number or you'll be rate limited. 5 is a good start. |
| store.concurrent_downloads | No | 1 | Number of concurrently downloaded images and videos. |
| store.album_filter | No | | Only download albums matching the filter |

## Run

Expand Down
3 changes: 3 additions & 0 deletions api.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (
"fmt"
"os"
"time"
"strings"

log "github.com/sirupsen/logrus"
)
Expand Down Expand Up @@ -115,6 +116,7 @@ func (w *Worker) saveImage(image albumImage, folder string) error {
return errors.New("unable to find valid image filename, skipping")
}
dest := fmt.Sprintf("%s/%s", folder, image.Name())
dest = strings.Replace(dest, "?", "_", -1)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not related to the current PR. It is, instead, related to #62
It's better to split the work, so that each one can have its own discussion

log.Debug(image.ArchivedUri)

ok, err := w.downloadFn(dest, image.ArchivedUri, image.ArchivedSize)
Expand All @@ -135,6 +137,7 @@ func (w *Worker) saveVideo(image albumImage, folder string) error {
return errors.New("unable to find valid video filename, skipping")
}
dest := fmt.Sprintf("%s/%s", folder, image.Name())
dest = strings.Replace(dest, "?", "_", -1)

if image.Processing {
if image.Status == "Preprocess" && image.SubStatus == "CanNotProcess" {
Expand Down
93 changes: 47 additions & 46 deletions doc.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,65 +1,66 @@
/*
Package smugmug implements the logic to perform a full
backup of a SmugMug account (images and videos).
Package smugmug implements the logic to perform a full
backup of a SmugMug account (images and videos).

The program loops over the images and videos of the user's
albums and saves them in the destination folder, replicating
the SmugMug paths.
The program loops over the images and videos of the user's
albums and saves them in the destination folder, replicating
the SmugMug paths.

You can run the app multiple times, all existing files will
be skipped if their sizes match.
You can run the app multiple times, all existing files will
be skipped if their sizes match.

Creating a binary with this package is as simple as:
Creating a binary with this package is as simple as:

package main
package main

import (
log "github.com/sirupsen/logrus"
"github.com/tommyblue/smugmug-backup"
)
import (
log "github.com/sirupsen/logrus"
"github.com/tommyblue/smugmug-backup"
)

func main() {
cfg, err := smugmug.ReadConf()
if err != nil {
log.WithError(err).Fatal("Configuration error")
}
func main() {
cfg, err := smugmug.ReadConf()
if err != nil {
log.WithError(err).Fatal("Configuration error")
}

wrk, err := smugmug.New(cfg)
if err != nil {
log.WithError(err).Fatal("Can't initialize the package")
}
wrk, err := smugmug.New(cfg)
if err != nil {
log.WithError(err).Fatal("Can't initialize the package")
}

if err := wrk.Run(); err != nil {
log.Fatal(err)
}
if err := wrk.Run(); err != nil {
log.Fatal(err)
}
}

The app reads its configuration from ./config.toml or $HOME/.smgmg/config.toml.
The app reads its configuration from ./config.toml or $HOME/.smgmg/config.toml.

The supported configuration keys/values are the following:
The supported configuration keys/values are the following:

[authentication]
username = "<SmugMug username>"
api_key = "<API Key>"
api_secret = "<API Secret>"
user_token = "<User Token>"
user_secret = "<User Secret>"
[authentication]
username = "<SmugMug username>"
api_key = "<API Key>"
api_secret = "<API Secret>"
user_token = "<User Token>"
user_secret = "<User Secret>"

[store]
destination = "<Backup destination folder>"
file_names = "{{.FileName}}"
[store]
destination = "<Backup destination folder>"
file_names = "{{.FileName}}"

All values can be overridden by environment variables, that have the following names:
All values can be overridden by environment variables, that have the following names:

SMGMG_BK_USERNAME = "<SmugMug username>"
SMGMG_BK_API_KEY = "<API Key>"
SMGMG_BK_API_SECRET = "<API Secret>"
SMGMG_BK_USER_TOKEN = "<User Token>"
SMGMG_BK_USER_SECRET = "<User Secret>"
SMGMG_BK_DESTINATION = "<Backup destination folder>"
SMGMG_BK_FILE_NAMES = "<Backup destination folder>"
SMGMG_BK_USERNAME = "<SmugMug username>"
SMGMG_BK_API_KEY = "<API Key>"
SMGMG_BK_API_SECRET = "<API Secret>"
SMGMG_BK_USER_TOKEN = "<User Token>"
SMGMG_BK_USER_SECRET = "<User Secret>"
SMGMG_BK_DESTINATION = "<Backup destination folder>"
SMGMG_BK_FILE_NAMES = "<Filename with template replacements>"
SMGMG_BK_ALBUM_FILTER = "<Only download albums matching the filter>"

All configuration values are required. They can be omitted in the configuration file
as long as they are overridden by environment values.
All configuration values are required. They can be omitted in the configuration file
as long as they are overridden by environment values.
*/
package smugmug
14 changes: 14 additions & 0 deletions smugmug.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"text/template"

Expand All @@ -28,6 +29,7 @@ type Conf struct {
ConcurrentAlbums int // number of concurrent albums analyzed via API calls
HTTPBaseUrl string // Smugmug API URL, defaults to https://api.smugmug.com
HTTPMaxRetries int // Max number of retries for HTTP calls, defaults to 3
AlbumFilter string // Only download albums matching the filter (currently just HasPrefx:)

username string
metadataFile string
Expand Down Expand Up @@ -59,6 +61,10 @@ func (cfg *Conf) overrideEnvConf() {
if os.Getenv("SMGMG_BK_FILE_NAMES") != "" {
cfg.Filenames = os.Getenv("SMGMG_BK_FILE_NAMES")
}

if os.Getenv("SMGMG_BK_ALBUM_FILTER") != "" {
cfg.AlbumFilter = os.Getenv("SMGMG_BK_ALBUM_FILTER")
}
}

func (cfg *Conf) validate() error {
Expand Down Expand Up @@ -134,6 +140,7 @@ func ReadConf(cfgPath string) (*Conf, error) {
ForceVideoDownload: viper.GetBool("store.force_video_download"),
ConcurrentDownloads: viper.GetInt("store.concurrent_downloads"),
ConcurrentAlbums: viper.GetInt("store.concurrent_albums"),
AlbumFilter: viper.GetString("store.album_filter"),
HTTPBaseUrl: viper.GetString("http.base_url"),
HTTPMaxRetries: viper.GetInt("http.max_retries"),
}
Expand Down Expand Up @@ -224,6 +231,13 @@ func (w *Worker) albumWorker(id int) {
log.Debugf("Quitting albumWorker %d", id)
return
}

filter := w.cfg.AlbumFilter
if len(filter) > 0 && !strings.HasPrefix(strings.ToLower(album.URLPath), strings.ToLower(filter)) { // Improve: use a real 'mathces' function instead of just HasPrefix
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since filter is a string, maybe using if string != "" is a bit more clear

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using lowercase here makes the strong assumption that the user wants a case-insensitive filter. Maybe this is the correct thing, maybe not. Using a regexp match can be more powerful here, although a little bit more expensive

log.Infof("Ignoring album: %s", album.URLPath)
continue
}

folder := filepath.Join(w.cfg.Destination, album.URLPath)

if err := createFolder(folder); err != nil {
Expand Down