![@aiborisov @mykyta_p Leaderboard API (REST) /players/<username>/score {"name": "Jane", "score": 100} /leaderboard/top/<n> [{"name": "Jane", "score": 100}, {"name": "John", "score": 50}, ...] @aiborisov @mykyta_p](https://image.slidesharecdn.com/latestdevoxxbelgiumbreakmeforslies-181115153146/75/Break-me-if-you-can-practical-guide-to-building-fault-tolerant-systems-with-examples-from-REST-and-gRPC-polyglot-stacks-23-2048.jpg)
Uploaded byAlex Borysov
Break me if you can: practical guide to building fault-tolerant systems (with examples from REST and gRPC polyglot stacks)
This document summarizes a presentation titled "Break Me If You Can: Practical Guide to Building Fault-Tolerant Systems" given at Devoxx Belgium in November 2018. The presentation discusses building fault tolerance into distributed systems through code and design patterns. It introduces concepts like faults, errors and failures, and provides an example architecture called "Dodging Geese" that incorporates techniques like timeouts and non-blocking calls to upstream services to handle potential faults and slowdowns. The presentation also covers monitoring and tracing distributed requests across services for observability.
Break me if you can: practical guide to building fault-tolerant systems (with examples from REST and gRPC polyglot stacks)
- 1. Break Me IfYou Can Practical Guide to Building Fault-tolerant Systems Devoxx Belgium, November 15, 2018 Alex Borysov, Software Engineer @ Google Mykyta Protsenko, Software Engineer @ Netflix
- 2. Who are we? AlexBorysov Software Engineer @Google Mykyta Protsenko Software Engineer @Netflix @aiborisov @mykyta_p
- 3.
- 4. Fault vs Errorvs Failure @aiborisov @mykyta_p
- 5. @aiborisov @mykyta_p Fault @aiborisov @mykyta_p incorrect internal state Picture by BobMcMillan. Public domain. See slide #180 for details.
- 6. @aiborisov @mykyta_p Error @aiborisov @mykyta_p visibly incorrect behaviour Picture by DavidGoehring. CC BY 2.0. See slide #180 for details.
- 7. @aiborisov @mykyta_p Failure @aiborisov @mykyta_p main functionality is broken Picture byCamerafiend. CC BY-SA 3.0. See slide #180 for details.
- 8. @aiborisov @mykyta_p RMS Titanic vsMiracle on the Hudson @aiborisov @mykyta_p Willy Stöwer. Public domain. See slide #180 for details. By Greg Lam Pak Ng. CC BY 2.0. See slide #181 for details.
- 9. @aiborisov @mykyta_p RMS Titanic @aiborisov @mykyta_p Fault: Hittingan iceberg Error: Water in the hull Failure: Sinking Willy Stöwer. Public domain. See slide #180 for details.
- 10. @aiborisov @mykyta_p Miracle on theHudson @aiborisov @mykyta_p Fault: Hitting geese at 859 m Error: Engines shut down No Failure! By Greg Lam Pak Ng. CC BY 2.0. See slide #181 for details.
- 11.
- 12.
- 13. @aiborisov @mykyta_p Fault Tolerance @aiborisov @mykyta_p Code andDesign Patterns Product-Driven Decisions Communication By Greg Lam Pak Ng. CC BY 2.0. See slide #181 for details.
- 14.
- 15. @aiborisov @mykyta_p Dodging Geese Architecture TOP-5 GeeseService Clouds Service Leaderboard Service API Gateway @aiborisov @mykyta_p See slides ##180, 181 for licensing details.
- 16. @aiborisov @mykyta_p Dodging Geese Architecture TOP-5 GeeseService Clouds Service Leaderboard Service API Gateway @aiborisov @mykyta_p
- 17. @aiborisov @mykyta_p Dodging Geese Architecture TOP-5 GeeseService Leaderboard Service API Gateway @aiborisov @mykyta_p Clouds Service
- 18. @aiborisov @mykyta_p Dodging Geese Architecture TOP-5 LeaderboardService API Gateway @aiborisov @mykyta_p Clouds Service Geese Service
- 19. @aiborisov @mykyta_p Dodging Geese Architecture GeeseService Clouds ServiceAPI Gateway @aiborisov @mykyta_p TOP-5 Leaderboard Service
- 20. @aiborisov @mykyta_p Dodging Geese Architecture TOP-5 GeeseService Clouds Service Leaderboard Service API Gateway @aiborisov @mykyta_p
- 21. @aiborisov @mykyta_p Dodging Geese Architecture TOP-5 GeeseService Clouds Service Leaderboard Service API Gateway @aiborisov @mykyta_p
- 22. @aiborisov @mykyta_p Dodging Geese Architecture TOP-5 GeeseService Clouds Service Leaderboard Service API Gateway @aiborisov @mykyta_p
- 23. @aiborisov @mykyta_p Leaderboard API (REST) /players/<username>/score {"name":"Jane", "score": 100} /leaderboard/top/<n> [{"name": "Jane", "score": 100}, {"name": "John", "score": 50}, ...] @aiborisov @mykyta_p
- 24. @aiborisov @mykyta_p gRPC Service Definitions @aiborisov @mykyta_p serviceGeeseService { // Return next line of geese. rpc GetGeese (GetGeeseRequest) returns (GeeseResponse); }
- 25. @aiborisov @mykyta_p gRPC Service Definitions @aiborisov @mykyta_p serviceGeeseService { // Return next line of geese. rpc GetGeese (GetGeeseRequest) returns (GeeseResponse); } service CloudsService { // Return next line of clouds. rpc GetClouds (GetCloudsRequest) returns (CloudsResponse); }
- 26. @aiborisov @mykyta_p service FixtureService { //Return next line of geese and clouds. rpc GetFixture (GetFixtureRequest) returns (FixtureResponse); } gRPC Gateway Service @aiborisov @mykyta_p
- 27. @aiborisov @mykyta_p service FixtureService { //Return next line of geese and clouds. rpc GetFixture (GetFixtureRequest) returns (FixtureResponse); } + = Fixture gRPC Gateway Service @aiborisov @mykyta_p
- 28. @aiborisov @mykyta_p public class FixtureServiceextends FixtureServiceImplBase { Gateway Fixture Service @aiborisov @mykyta_p
- 29. @aiborisov @mykyta_p Gateway Fixture Service GeeseService Clouds ServiceAPI Gateway @aiborisov @mykyta_p
- 30. @aiborisov @mykyta_p Gateway Fixture Service CloudsServiceAPI Gateway @aiborisov @mykyta_p Geese Service
- 31. @aiborisov @mykyta_p Gateway Fixture Service CloudsServiceAPI Gateway @aiborisov @mykyta_p Geese Service
- 32. @aiborisov @mykyta_p Gateway Fixture Service API Gateway @aiborisov @mykyta_p GeeseService Clouds Service
- 33. @aiborisov @mykyta_p Gateway Fixture Service API Gateway @aiborisov @mykyta_p GeeseService Clouds Service
- 34. @aiborisov @mykyta_p @aiborisov @mykyta_p Fixture Latency = GeeseLatency + Clouds Latency
- 35. @aiborisov @mykyta_p @aiborisov @mykyta_p Non-Blocking Calls Don’t block Sendrequests in parallel Combine results when ready
- 36. @aiborisov @mykyta_p public class FixtureServiceextends FixtureServiceImplBase { Gateway Service Implementation @aiborisov @mykyta_p private final GeeseServiceFutureStub geeseClient = ...; private final CloudsServiceFutureStub cloudsClient = ...;
- 37. @aiborisov @mykyta_p public class FixtureServiceextends FixtureServiceImplBase { Gateway Service Implementation @aiborisov @mykyta_p private final GeeseServiceFutureStub geeseClient = ...; private final CloudsServiceFutureStub cloudsClient = ...; @Override public void getFixture(GetFixtureRequest request, StreamObserver<FixtureResponse> response) { ListenableFuture<GeeseResponse> geese = geeseClient.getGeese(toGeese(request)); ListenableFuture<CloudsResponse> clouds = cloudsClient.getClouds(toClouds(request)); ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds = Futures.allAsList(geese, clouds); ...
- 38. @aiborisov @mykyta_p public class FixtureServiceextends FixtureServiceImplBase { Gateway Service Implementation @aiborisov @mykyta_p private final GeeseServiceFutureStub geeseClient = ...; private final CloudsServiceFutureStub cloudsClient = ...; @Override public void getFixture(GetFixtureRequest request, StreamObserver<FixtureResponse> response) { ListenableFuture<GeeseResponse> geese = geeseClient.getGeese(toGeese(request)); ListenableFuture<CloudsResponse> clouds = cloudsClient.getClouds(toClouds(request)); ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds = Futures.allAsList(geese, clouds); ...
- 39.
- 40.
- 41.
- 42. @aiborisov @mykyta_p public class FixtureServiceextends FixtureServiceImplBase { ... Gateway Service Implementation @aiborisov @mykyta_p @Override public void getFixture(GetFixtureRequest request, StreamObserver<FixtureResponse> response) { ListenableFuture<GeeseResponse> geese = geeseClient.getGeese(toGeese(request)); ListenableFuture<CloudsResponse> clouds = cloudsClient.getClouds(toClouds(request)); ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds = Futures.allAsList(geese, clouds); ...
- 43. @aiborisov @mykyta_p public class FixtureServiceextends FixtureServiceImplBase { ... Gateway Service Implementation @aiborisov @mykyta_p @Override public void getFixture(GetFixtureRequest request, StreamObserver<FixtureResponse> response) { ListenableFuture<GeeseResponse> geese = geeseClient.withDeadlineAfter(500, MILLISECONDS).getGeese(toGeeseRequest(request)); ListenableFuture<CloudsResponse> clouds = cloudsClient.withDeadlineAfter(500, MILLISECONDS).getClouds(toCloudsRequest(request)); ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds = Futures.allAsList(geese, clouds); ...
- 44. @aiborisov @mykyta_p @Override public void getFixture(GetFixtureRequestrequest, StreamObserver<FixtureResponse> response) { ListenableFuture<GeeseResponse> geese = geeseClient.withDeadlineAfter(500, MILLISECONDS).getGeese(toGeeseRequest(request)); ListenableFuture<CloudsResponse> clouds = cloudsClient.withDeadlineAfter(500, MILLISECONDS).getClouds(toCloudsRequest(request)); ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds = Futures.allAsList(geese, clouds); ... public class FixtureService extends FixtureServiceImplBase { ... Gateway Service Implementation @aiborisov @mykyta_p
- 45. @aiborisov @mykyta_p REST: Non-Blocking Calls CompletableFuture<List<LeaderboardEntry>>leaderboard = httpClient .get().uri("/top/5") .exchange() .timeout(Duration.ofMillis(500)) .flatMap(cr -> cr.bodyToMono(...)) .toFuture(); @aiborisov @mykyta_p
- 46. @aiborisov @mykyta_p REST: Non-Blocking Callswith Timeout CompletableFuture<List<LeaderboardEntry>> leaderboard = httpClient .get().uri("/top/5") .exchange() .timeout(Duration.ofMillis(500)) .flatMap(cr -> cr.bodyToMono(...)) .toFuture(); @aiborisov @mykyta_p
- 47.
- 48.
- 49.
- 50.
- 51. @aiborisov @mykyta_p @aiborisov @mykyta_p Observability: gRPC Monitoring: QPS, latency,errors, ... // OpenCensus RpcViews.registerAllViews();
- 52. @aiborisov @mykyta_p @aiborisov @mykyta_p Tracing: gRPC GrpcTracing grpcTracing= GrpcTracing.create(...); ManagedChannelBuilder ... .intercept(grpcTracing.newClientInterceptor()) .build() ; ServerBuilder.forPort(8080) ... .intercept(grpcTracing.newServerInterceptor()) .build();
- 53. @aiborisov @mykyta_p @aiborisov @mykyta_p Tracing: gRPC GrpcTracing grpcTracing= GrpcTracing.create(...); ManagedChannelBuilder ... .intercept(grpcTracing.newClientInterceptor()) .build(); ServerBuilder.forPort(8080) ... .intercept(grpcTracing.newServerInterceptor()) .build();
- 54. @aiborisov @mykyta_p @aiborisov @mykyta_p Tracing: REST build.gradle: dependencies { compile'...:spring-cloud-sleuth-zipkin' compile '...:spring-cloud-starter-sleuth' ... } application.properties: spring.zipkin.baseUrl=http://zipkin:9411/ spring.sleuth.sampler.probability=1.0 spring.sleuth.web.enabled=true
- 55.
- 56.
- 57. @aiborisov @mykyta_p @aiborisov @mykyta_p Clouds are slow Geeseare fast Entire call fails
- 58. @aiborisov @mykyta_p ListenableFuture<GeeseResponse> geese = geeseClient..getGeese(toGeese(request)); ListenableFuture<CloudsResponse>clouds = cloudsClient.getClouds(toClouds(request)); ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds = Futures.allAsList(geese, clouds); ... @aiborisov @mykyta_p Partial Degradation
- 59. @aiborisov @mykyta_p @aiborisov @mykyta_p Partial Degradation ListenableFuture<GeeseResponse> geese= geeseClient..getGeese(toGeese(request)); ListenableFuture<CloudsResponse> clouds = cloudsClient.getClouds(toClouds(request)); ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds = Futures.successfulAsList(geese, clouds); ...
- 60.
- 61. @aiborisov @mykyta_p @aiborisov @mykyta_p Some L-board callsfail L-board latency is low Scores disappear
- 62. @aiborisov @mykyta_p CompletableFuture<List<Leaderboard>> request() { returnhttpClient .get().uri("/top/5").exchange() .timeout(Duration.ofMillis(500)) .flatMap(...).toFuture(); } @aiborisov @mykyta_p Retries: REST
- 63. @aiborisov @mykyta_p CompletableFuture<List<Leaderboard>> request() { returnhttpClient .get().uri("/top/5").exchange() .timeout(Duration.ofMillis(500)) .flatMap(...).toFuture(); } RetryPolicy RETRY_POLICY = new RetryPolicy() .retryOn(IOException.class) .withMaxRetries(MAX_RETRIES); CompletableFuture<List<Leaderboard>> top5 = Failsafe.with(RETRY_POLICY) ... .future(this::httpRequest); @aiborisov @mykyta_p Retries: REST
- 64.
- 65.
- 66. @aiborisov @mykyta_p @aiborisov @mykyta_p Retry slow calls? Retryfailed calls? Retry network faults?
- 67.
- 68.
- 69.
- 70.
- 71.
- 72.
- 73.
- 74. @aiborisov @mykyta_p @aiborisov @mykyta_p High 99%ile latency 100requests Error probability?
- 75. @aiborisov @mykyta_p @aiborisov @mykyta_p High 99%ile latency 100requests Error probability: 1 – 0.99^100 = 63%
- 76.
- 77.
- 78.
- 79. @aiborisov @mykyta_p Tail-Tolerance @aiborisov @mykyta_p Request 200 ms deadline ↓100 ms Request Fastest Response
- 80. @aiborisov @mykyta_p High 99%ile latency 100requests @aiborisov @mykyta_p Request Hedging
- 81. @aiborisov @mykyta_p High 99%ile latency 100requests Error probability: 63% x 0.01 < 1% @aiborisov @mykyta_p Request Hedging
- 82. @aiborisov @mykyta_p Channel geeseChannel =ManagedChannelBuilder .forAddress(geeseHost, geesePort) .enableRetry() .maxHedgedAttempts(MAX_HEDGES) .build(); GeeseServiceFutureStub geeseStub = GeeseServiceGrpc .newFutureStub(geeseChannel); @aiborisov @mykyta_p Hedging in gRPC (soon)
- 83. @aiborisov @mykyta_p Channel geeseChannel =ManagedChannelBuilder .forAddress(geeseHost, geesePort) .enableRetry() .maxHedgedAttempts(MAX_HEDGES) .build(); GeeseServiceFutureStub geeseStub = GeeseServiceGrpc .newFutureStub(geeseChannel); @aiborisov @mykyta_p Hedging in gRPC (soon)
- 84.
- 85.
- 86. @aiborisov @mykyta_p @aiborisov @mykyta_p High mean latency 100requests Error probability?
- 87. @aiborisov @mykyta_p @aiborisov @mykyta_p High mean latency 100requests Error probability: 1 – 0.50^100 = 99.99...%
- 88. @aiborisov @mykyta_p CircuitBreaker CIRCUIT_BREAKER = newCircuitBreaker() .withFailureThreshold(3, 5); CompletableFuture<...> top5 = Failsafe .with(CIRCUIT_BREAKER) .with(RETRY_POLICY) ... .future(this::httpRequest); @aiborisov @mykyta_p Circuit Breaker
- 89. @aiborisov @mykyta_p @aiborisov @mykyta_p Error Handling 100% ErrorFail Fast Intermittent Slow Hedging Intermittent Fast Retry Fallback✚
- 90. @aiborisov @mykyta_p @aiborisov @mykyta_p Error Handling 100% ErrorFail Fast Intermittent Slow Hedging Intermittent Fast Retry Fallback✚
- 91.
- 92.
- 93. @aiborisov @mykyta_p Deadlines API Gateway @aiborisov @mykyta_p See slides ##180,181 for licensing details.
- 94.
- 95.
- 96. @aiborisov @mykyta_p Deadlines API Gateway @aiborisov @mykyta_p Spent 120 ms →Spent 90 ms Deadline 200 ms → X
- 97. @aiborisov @mykyta_p Deadlines API Gateway @aiborisov @mykyta_p Spent 120 ms →Spent 90 ms Deadline 200 ms → X →
- 98.
- 99. @aiborisov @mykyta_p Deadline 80 ms DeadlinesPropagation API Gateway @aiborisov @mykyta_p Deadline 200 ms → Spent 120 ms →
- 100. @aiborisov @mykyta_p Deadline 80 ms DeadlinesPropagation API Gateway @aiborisov @mykyta_p Spent 120 ms → Spent 90 ms Deadline 200 ms → X
- 101. @aiborisov @mykyta_p Deadline 80 ms DeadlinesPropagation API Gateway @aiborisov @mykyta_p Spent 120 ms → Spent 90 ms Deadline -10 ms Deadline 200 ms → X
- 102.
- 103.
- 104. @aiborisov @mykyta_p new ConcurrencyLimitServletFilter( new ServletLimiterBuilder() .partitionByHeader("GEESE_TYPE", c-> c.assign("premium", 0.9) .assign("free", 0.1)) .limiter(l -> l.limit( newBuilder() .initialLimit(1000)...); @aiborisov @mykyta_p REST
- 105. @aiborisov @mykyta_p new ConcurrencyLimitServletFilter( new ServletLimiterBuilder() .partitionByHeader("GEESE_TYPE", c-> c.assign("premium", 0.9) .assign("free", 0.1)) .limiter(l -> l.limit( newBuilder() .initialLimit(1000)...); @aiborisov @mykyta_p REST
- 106. @aiborisov @mykyta_p var limiter = newGrpcServerLimiterBuilder() .partitionByHeader(GEESE_TYPE) .partition("premium", 0.9) .partition("free", 0.1) .limiter(l -> l.limit( newBuilder() .initialLimit(1000)...); ConcurrencyLimitServerInterceptor .newBuilder(limiter).build(); @aiborisov @mykyta_p gRPC: Server
- 107. @aiborisov @mykyta_p var limiter = newGrpcServerLimiterBuilder() .partitionByHeader(GEESE_TYPE) .partition("premium", 0.9) .partition("free", 0.1) .limiter(l -> l.limit( newBuilder() .initialLimit(1000)...); ConcurrencyLimitServerInterceptor .newBuilder(limiter).build(); @aiborisov @mykyta_p gRPC: Server
- 108. @aiborisov @mykyta_p new GrpcClientLimiterBuilder() .limit( newBuilder() .initialLimit(1000).build()) .blockOnLimit(false) //fail-fast .build(); @aiborisov @mykyta_p gRPC: Client
- 109.
- 110.
- 111.
- 112. @aiborisov @mykyta_p Monitoring @aiborisov @mykyta_p APM Service metrics Distributed tracing Business metrics Picture by AlexBorysov. CC BY 2.0. See slide #180 for details.
- 113. @aiborisov @mykyta_p @aiborisov @mykyta_p Code and Design Timeouts/ Deadline Propagation Retries / Hedging Proper Fallbacks Concurrency Limits Load Shedding Observability
- 114. @aiborisov @mykyta_p @aiborisov @mykyta_p Request for eachresponse Requests don’t change
- 115.
- 116.
- 117.
- 118. @aiborisov @mykyta_p service GeeseService { rpcGetGeese (GetGeeseRequest) returns (GeeseResponse); } service CloudsService { rpc GetClouds (GetCloudsRequest) returns (CloudsResponse); } @aiborisov @mykyta_p gRPC Streaming
- 119. @aiborisov @mykyta_p service GeeseService { rpcGetGeese (GetGeeseRequest) returns (stream GeeseResponse); } service CloudsService { rpc GetClouds (GetCloudsRequest) returns (stream CloudsResponse); } @aiborisov @mykyta_p gRPC Streaming
- 120.
- 121. @aiborisov @mykyta_p @aiborisov @mykyta_p Server faster thanclient Client cannot keep up
- 122. @aiborisov @mykyta_p Too Many StreamingResponses @aiborisov @mykyta_p GeeseRequest
- 123. @aiborisov @mykyta_p Too Many StreamingResponses @aiborisov @mykyta_p GeeseRequest X
- 124.
- 125.
- 126.
- 127.
- 128.
- 129.
- 130.
- 131.
- 132.
- 133.
- 134. @aiborisov @mykyta_p Door Capacity @aiborisov @mykyta_p Why didn’tRose make room for Jack on the door? Willy Stöwer. Public domain. See slide #180 for details.
- 135. @aiborisov @mykyta_p Door Capacity @aiborisov @mykyta_p Why didn’tRose make room for Jack on the door? “ The answer is very simple because it says on page 147 that Jack dies “ James Cameron Willy Stöwer. Public domain. See slide #180 for details.
- 136.
- 137.
- 138.
- 139.
- 140. @aiborisov @mykyta_p Prescaling @aiborisov @mykyta_p See slides ##180,182 for licensing details.
- 141.
- 142.
- 143.
- 144. @aiborisov @mykyta_p Free and PremiumOutage Free Premium $ $
- 145.
- 146.
- 147. @aiborisov @mykyta_p @aiborisov @mykyta_p Bulkheads By Request Type ByClient Priority By Region By Availability Zone etc
- 148.
- 149.
- 150.
- 151.
- 152. @aiborisov @mykyta_p Prober TOP-5 API Gateway @aiborisov @mykyta_p See slides ##180,182 for licensing details.
- 153.
- 154.
- 155.
- 156.
- 157.
- 158.
- 159.
- 160.
- 161.
- 162.
- 163.
- 164.
- 165.
- 166. @aiborisov @mykyta_p Postmortems @aiborisov @mykyta_p Blameless Constructive Social See slides ##189,182, 183 for licensing details.
- 167.
- 168. @aiborisov @mykyta_p @aiborisov @mykyta_p Learn from Failure Blamelesspostmortems Alert playbooks Incident knowledge base
- 169.
- 170. @aiborisov @mykyta_p Libraries and Tools @aiborisov @mykyta_p Demo:github.com/break-me-if-you-can Failsafe: github.com/jhalterman/failsafe Observability: opencensus.io, opentracing.io Prober: cloudprober.org Concurrency Limits: github.com/Netflix/concurrency-limits
- 171. @aiborisov @mykyta_p Demo UI @HalloGene_ Yevgen Golubenko Twitter:@HalloGene_ github.com/HalloGene Picture by Yevgen Golubenko. Also see slide #183 for licensing details.
- 172.
- 173. @aiborisov @mykyta_p @aiborisov @mykyta_p Fault-Tolerance Code & DesignPatterns Product decisions Communication culture
- 174.
- 175.
- 176. @aiborisov @mykyta_p Please Break Me! RateUs If you can
- 177. @aiborisov @mykyta_p Please Break Me! RateUs If you enjoyed the talk Or give feedback If you can
- 178. @aiborisov @mykyta_p Please Break Me! RateUs If you enjoyed the talk Or give feedback If you can 5 STARS!
- 179.
- 180. @aiborisov @mykyta_p Images and Licensing Imagesof geese, clouds, pilots, plane, arrows, cup, airport traffic control tower are property of Mykyta Protsenko and Alex Borysov, if not stated otherwise (see below). All Rights Reserved. Other images used: Slide #5: commons.wikimedia.org/wiki/File:FEMA_-_16381_-_Photograph_by_Bob_McMillan_taken_on_09-28-2005_in_Texas.jpg - Picture by Bob McMillan, the US federal government work, public domain Slide #6: www.flickr.com/photos/carbonnyc/3290528875 - Picture by David Goehring. Attribution 2.0 Generic (CC BY 2.0): creativecommons.org/licenses/by/2.0 - changes were made Slide #7: www.flickr.com/photos/carbonnyc/3290528875 - Picture by Camerafiend. Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0): creativecommons.org/licenses/by-sa/3.0/deed.en - no changes were made Slides ##8, 9, 134, 135: commons.wikimedia.org/wiki/File:Titanic_sinking,_painting_by_Willy_St%C3%B6wer.jpg - Willy Stöwer. Public domain work of art
- 181. @aiborisov @mykyta_p Images and Licensing Slides##8, 10, 13: www.flickr.com/photos/22608787@N00/3200086900 - Picture y Greg Lam Pak Ng. Attribution 2.0 Generic (CC BY 2.0): creativecommons.org/licenses/by/2.0 - no changes were made Slides ##15-22, 29-33, 67, 76-79, 93-101, 115-117, 122-128, 136-140, 143-146, 151-152: - Blue Game Boy Color by kure: piq.codeus.net/picture/31994/Blue-Game-Boy-Color - Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0 - changes were made Slides ##93-101: - The Sun by Vinicius615: piq.codeus.net/picture/191706/The-Sun - Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0 - changes were made Slide #112: - Picture by Alex Borysov. Attribution 2.0 Generic (CC BY 2.0): creativecommons.org/licenses/by/2.0
- 182. @aiborisov @mykyta_p Images and Licensing Slide#140: piq.codeus.net/picture/254492/CVsantahat - Santa hat for CommanderVideo, CVsantahat by anonymous - Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0 - no changes were made Slide #152: piq.codeus.net/picture/423109/UFO - UFO by anonymous - Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0 - no changes were made Slides #166, 167: piq.codeus.net/picture/334023/beer - beer by Investa - Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0 - changes were made
- 183. @aiborisov @mykyta_p Images and Licensing Slides#166, 167: piq.codeus.net/picture/444498/Beer-Bottle - Beer Bottle by jacklrj - Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0 - changes were made Slide #171: https://piq.codeus.net/picture/330338/Deal-With-It - Deal With It by Shiro - Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0 - changes were made