개발
-
[Kafka] - 1. What is Apache Kafka개발/Kafka 2022. 8. 3. 21:24
Apache Kafka Linked-in에서 일 4.5조개의 Data를 처리하기 위하여 만든 솔루션 Data in motion platform = Event-Stream Platform Event란? 비즈니스에서 일어나는 모든 일 청구서 발행, 배송 물건의 위치 정보, 택시 GPS, Sensor 온도 등등.. Event가 발생함으로써 특징 Bussiness 모든 영역에서 광범위하게 발생 대용량의 Data가 발생한다 -> Message Queue로는 처리 속도를 잡을 수 없다. -> Event-Streaming Data의 필요성 Kafka는 이를 처리 할 수 있도록 만들었다. 특징 1. Event-Stream을 안전하게 전송(Publish&Subscribe) 2. Write to Disk 3. Proces..
-
[Docker] - Docker-compose개발/Docker 2022. 8. 3. 17:24
Docker-compose 개요 Docker compose란, 여러 컨테이너 서비스를 구축, 실행하는 순서를 자동으로 하여, 관리를 간단히 하는 기능이다. Docker compose에서는 docker-compose.yaml(docker-compose.yml은 우선순위가 후순위) 파일을 실행, 설정을 읽어 들여 모든 컨테이너 서비스를 실행 Docker-compose Process Document를 보게되면, 아래와 같이 정의되어 있다. Define your app’s environment with a Dockerfile so it can be reproduced anywhere. Define the services that make up your app in docker-compose.yml so they..
-
[Airflow] - Concept개발/Airflow 2022. 8. 3. 15:40
Airflow Default Concept 1. Operator An Operator is conceptually a template for a predefined Task, that you can just define declaratively inside your DAG For a list of all core operators(built-in) BashOperator - executes a bash command PythonOperator - calls an arbitrary Python function EmailOperator - sends an email with DAG("my-dag") as dag: ping = SimpleHttpOperator(endpoint="http://example.co..
-
[Airflow] - what is Airflow개발/Airflow 2022. 7. 30. 00:56
Airflow Airflow is a platform to programmatically author, schedule and monitor workflows. Use Airflow to author workflows as Directed Acyclic Graphs (DAGs) of tasks The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Airflow is not a data streaming(data load like image,youtube ) solution. Principles 1. Dynamic Airflow pipelines are configu..
-
[Python] - Closure개발/Python 2022. 7. 28. 20:59
Closure 1. Nested functions A function that is defined inside another function is known as a nested function. Nested functions are able to access variables of the enclosing scope. In Python, these non-local variables can be accessed only within their scope and not outside their scope. But Closure can be access non-local variables #nested function example def nested_func(text): temp = text def te..
-
[Python] - First class functions개발/Python 2022. 7. 28. 17:55
1. First class Obejct Define 1. You can store them in Variable , Data Structure(list,dictionary .. ) 2. You can pass the function as a parameter to another function 3. You can return the function from a function. 2. First class functions Python support first class functions, it means function is instance of Object. First class functions includes First class Object properties. First class functio..
-
[Python] - Generator개발/Python 2022. 7. 28. 16:38
Generator Generator functions allow you to declare a function that behaves like an iterator, it can be used in a for loop. Generator call like iterator whenever you need 1. yield if use normal function 'return' get value , but Generator use 'yield' yield can be used to control the iteration behaviour of a loop instead of building an array containing all the values and returning them all at once,..